UX Case Study: Building AI Reporting tool for asset managers
Strategic Feature AI Design for IBM
Defining Solution-neutral problem statement and requirements.
1. Design Problem Description
Objectives:
Real-time Information Access: Enable the supervisor to access machine-specific information instantly and effortlessly using the HMD.
Enhanced Decision-Making: Empower the supervisor to make informed decisions on machine utilization and workforce allocation based on workload and expertise.
Seamless Integration: Ensure seamless integration of the AI tool with existing shop floor processes and workflows.
User Interface Design: Design an intuitive and user-friendly interface on the HMD that presents information clearly and allows for easy navigation.
Training and Adoption: Facilitate training and adoption of the tool among supervisors, ensuring they can effectively utilize its capabilities.
Constraints:
Technological Limitations: The design must work within the constraints of current optical see-through HMD technology, including display resolution, processing power, and battery life.
Compatibility: Ensure compatibility with existing machines and systems on the shop floor, including data interfaces and communication protocols.
Safety and Ergonomics: Address safety concerns related to the use of HMDs in industrial environments, such as visibility, distraction, and comfort during prolonged use.
Data Security: Implement robust data security measures to protect sensitive machine and workforce information accessed through the HMD.
Cost: Consider the cost implications of implementing and maintaining the AI-assisted tool, including hardware, software development, and training expenses.
Design Space:
Hardware Selection: Explore different models of optical see-through HMDs available in the market to choose one that balances performance, comfort, and cost-effectiveness.
Software Development: Develop AI algorithms and software applications that can efficiently analyse and present machine-specific information in real-time on the HMD.
User Interface Design: Design an intuitive graphical user interface (GUI) for the HMD that allows supervisors to navigate through machine data seamlessly and make decisions quickly.
Integration with Existing Systems: Plan and execute the integration of the AI tool with existing shop floor systems, ensuring minimal disruption to ongoing operations.
Training and Support: Develop training materials and provide ongoing support to supervisors to ensure they are proficient in using the AI tool effectively.
Reasons for choosing FAST daigram for Information architecture:
It clarifies the dependencies between different functions.
handles the complexity of interactions between machines, workers, and AI systems by expanding functions as needed to accommodate new requirements.
It allows for iterative refinement. As feedback is received (e.g., from supervisors or workers), new functions or optimizations can be added, such as enhancing the decision-making process or improving the user interface for better real-time data interpretation.
Based on FAST Diagram I derived the chosen concept using Morphological chart :
Then chosen concepts were evaluated for validation on various parameters:
Concept 1 was chosen with the highest weighted value of 129, because :
Human Control to Mitigate Risks: Critical and high-risk operations involve human oversight, allowing supervisors to manage complex, ambiguous, or high-impact decisions.
Full Automation for Low-Risk Areas: Routine tasks and processes are fully automated where human intervention adds little value, speeding up operations without sacrificing safety or compliance.
Flexible Supervisor Interventions: Automation enhances efficiency, while supervisors retain the ability to intervene, adjust, or override when necessary to ensure the best outcomes in critical scenarios.
Chosen concept solution (detailed) :
Verification cross-reference matrix (VCRM model)
Automation strategy :
Evaluation criteria applied to the automation strategy
Automation strategy (revised)
After applying the Primary user-centric evaluation criteria and secondary system-centred evaluation criteria , I reconsidered the choices of types and levels of automation to introduce more risk mitigation and human control for a mixed initiative design.
I arrived at final types and levels of automation illustrated in the table below.
Mixed Initiative design principles applied to the automation strategy :
User validation done through Usser testing :
Challenges :
Step 5: Interpretability issues arising in the system
Step 6: Issues around sharing of control and user agency.
II. Prompting Automation
Step 7: System boundary and an analysis of the key risks in the system
1. Defining System Boundary
Motivation : Ensure focused, effective operation and seamless integration with existing processes.
Approach:
Clearly define included and excluded elements to enhance functionality and prevent overlap.
Ensure the system operates within its intended scope, integrates smoothly with existing processes, and avoids redundancy, thereby enhancing overall operational efficiency and clarity.
2. System Mapping
System Mapping Method used :
I chose the system diagram combined with process lane diagram to represent the system mapping because :
1. Provides clear, step-by-step visual representation of the processes involved, from monitoring machinery to reallocating workers. This helps in understanding and managing complex workflows.
2. Highlights decision points and branching logic (e.g., failure detected or not), which helps in pinpointing how different scenarios are handled.
3. Helps identify bottlenecks or inefficiencies in the process, enabling targeted improvements.
4. Clarifies the roles and responsibilities of different system parts and stakeholders, improving coordination.
3. Determining an Acceptable Level of Risk and Risk Assessment
I have chosen the SWIFT (What-if) risk assessment method for assessing risk. An acceptable level of risk will be low to medium for the AI system as the risks can be mitigated by machine learning, feedback loops and upgradation of AI algorithm and training data. Risks which are high, needs to be mitigated to bring them to acceptable level of risk.
Analysis and retrospection
I have chosen the SWIFT (What-if) risk assessment method for assessing risk. An acceptable level of risk will be low to medium for the AI system as the risks can be mitigated by machine learning, feedback loops and upgradation of AI algorithm and training data. Risks which are high, needs to be mitigated to bring them to acceptable level of risk.
1. Is your system boundary truly encompassing all relevant factors, including policies and regulation?
Yes , the system boundary addresses key technical and operational aspects of the automation system, such as machinery monitoring, failure detection, worker allocation, and production oversight.
It also includes safety and compliance monitoring, which aligns with policy adherence and regulatory requirements with specific focus on integrating and enforcing detailed policies and regulations, not just through compliance tools but through continuous monitoring and updates.
Ensuring that these regulatory elements are fully embedded and regularly reviewed within all relevant modules will help in maintaining comprehensive adherence to industry standards and legal requirements.
2. What hazards did you identify? Were there hazards that were not exposed?
Hazards identified :
Potential for machinery breakdown, which could endanger workers.
Faulty machinery may produce defective products.
increased operational pressure when the AI system encounters a critical issue and fails during peak production times.
Misalignment of AI system with current production needs as production needs changes with new requirements.
Delayed detection of faults or failures
Missed alerts,
incorrect responses.
Yes , there are hazards which were not exposed :
Inadequate protection of sensitive data : It’s not exposed when adequate security protocols and systems are in place.
Poor integration with existing systems or processes :
It's not exposed as continuous training of AI system through feedback and machine learning can help ensure there is seamless integration.
3. Was your choice of system mapping technique useful for assessing risks later?
Yes, combining a system diagram with a process lane diagram is highly useful for assessing risks later as –
It helped in finding where issues might occur, such as communication gaps between components or process handoffs that could introduce errors.
Combining both diagrams allows for a thorough risk assessment by highlighting both technical and procedural risks that might not be evident when only one diagram is used.
Process diagram helps assess how failures in one process or lane can affect others and the overall workflow. while system diagram helps in understanding the impact of component failures on the overall system.
Process diagram helps in continuously evaluating process efficiency and adherence to procedures. While system diagram is useful for identifying areas to monitor for system performance and failure points. Together they help facilitate ongoing risk management by providing tools to monitor and address both system and process-related risks as the system evolves.
4. Was your choice of risk assessment method useful for assessing the risks in your system?
Yes, it was useful to use SWIFT method in combination with FMEA method , for risk assessment as –
It helps ensure that all possible scenarios are considered, promoting comprehensive risk assessment and uncovering issues that might otherwise be missed.
As the AI system evolves, new risks may emerge. SWIFT helps address ongoing risk assessment and adaptation to new scenarios, ensuring that risk management remains relevant as the system develops..
We can regularly apply SWIFT as the AI system is updated or new features are added, ensuring that emerging risks are addressed.
5. How would you redesign your automation function to reduce the level of risk?
I would introduce error handling and recovery processes to address unexpected situations.
Include automatic rollback and recovery procedures to minimize downtime.
Implement self-checks and diagnostic algorithms to detect and correct issues in real-time.
Apply predictive analytics to foresee potential failures and schedule proactive maintenance.
Step 10: Strategy for managing the risks and governance issues of a human-AI system
Approach :
Built to be proactive, dynamic, and lifecycle-integrated.
Focuses on continuous risk assessment, flexible governance structures, enhanced human-AI collaboration and commitment to continuous improvement, to ensure the AI system operates safely, ethically, and effectively.
Addresses current risks as well as predicts and adapts to future challenges, making the human-AI system resilient and sustainable over time.
Step 11: Heuristic evaluation and design implementation
