In August of 2024, our Developer Evangelist, Anna Maruyama invited Peaks Media to the Sony Shinagawa Prefecture headquarters for an intimate look at the origin story of AITRIOS, as well as a feature of its capabilities and offerings.
The in-depth discussion provides the ultimate look into AITRIOS, why it was established by Sony Semiconductor Solutions, how it works, and how this technology is becoming democratized for widespread adoptability and scalability of sensing, edge, and vision AI.
The following interview, translated from Japanese, has been published in association with PEAKS MEDIA . Read the original article in Japanese, here: https://www.peaks-media.com/8287/
SONY'S AITRIOS OPENS UP A NEW ERA OF VISION SENSING WITH SONY'S EDGE AI SENSING PLATFORM AITRIOS IN THE LOOMING "DATA EXPLOSION ERA"
From generative AI to deep learning, our daily lives are being drastically changed by AI.
For industry, AI will become a driver of future innovation and a driving force for the creation of new business models.
This time, with the theme of social implementation of AI, we covered the efforts of Sony Semiconductor Solutions, which provides services using an edge AI sensing platform called AITRIOS, with the aim of providing new value and contributing to solving problems to make everyday experiences more comfortable, convenient, and safe.
PROFILE:
Anna Maruyama, System Solutions Division, Sony Semiconductor Solutions
COMPANY: Sony Semiconductor Solutions, AITRIOS™
In 2014, Anna joined a major U.S. IT company. As a salesperson, she oversaw the development of new customers for mid-sized and major companies and contributed to solving their business issues and achieving their business goals through technology. In 2023, she moved to Sony Semiconductor Solutions, where she is currently responsible for AITRIOS' business development for developers around the world as an Evangelist at AITRIOS. Utilizing her experience in community launch and management at a major U.S. IT company and the American Chamber of Commerce in Japan, Anna is also engaged in user community development centered on AITRIOS.
Aiming to break away from cloud dependence
Sony has been a world leader in the field of image sensors for many years, but could you tell us about the background and background that led to the development and commercialization of AITRIOS?
Semiconductor "image sensors," which play a role like the "retina" of the human eye, are installed in various products around us, such as digital cameras, smartphones, and security cameras.
Since the introduction of image sensors more than 40 years ago, Sony has continued to innovate and contribute to society through its incorporation into its products. In addition to the application of "imaging" to beautifully capture the moment in front of us, we are accelerating the development of the "sensing" field as a unique initiative. This system acquires the necessary data from images taken by image sensors and uses them for various purposes, such as identification and detection by robots, industrial cameras, and in-vehicle cameras.
As AI becomes more widespread in the future, we launched AITRIOS because we believe that our next goal is to encourage our partner companies to work on the development of more solutions using image sensors for sensing applications.
With the spread of AI, why was AITRIOS necessary?
First, many services using IoT devices are currently widespread, and the cloud is often used as an infrastructure to support such services. However, excessive reliance on the cloud has created many challenges, such as increased power consumption in data centers, high network loads, and increased communication costs.
To cope with this situation, which is said to be the "data explosion era", there is an urgent need to reduce the amount of data flowing into the cloud through the network. Therefore, it is necessary to build a system that distributes the load, such as performing certain processing on the edge side such as IoT devices, rather than processing data only on the cloud side.
This challenge is particularly evident in vision sensing, which deals with very data-intensive materials such as images and videos. When acquiring the necessary data from the captured image, the amount of data in the image is very large compared to the amount of data from IoT sensors that detect other temperatures and humidity, and it is difficult to process it even with a computer, so the development of solutions using vision sensing is generally considered to be a high hurdle.
To prevent the explosive growth of data processed in the cloud, distributed processing at the edge and in the cloud is required. What are some of the difficulties of vision sensing using images other than the large amount of data involved?
With the human eye, for example, you can see that an apple is an apple even if you look at it in a bright or dark place.
However, when a computer tries to recognize an image of an apple taken with a camera, the apple may not look like an apple due to the difference in brightness and the way the light hits it. In other words, it may be difficult to recognize that they are the same depending on how they look, and this is a high hurdle in AI projects that utilize images (*hereinafter referred to as "vision").
Normally, the vision from the image sensor is transmitted to the cloud where the inference processing is running, and it is processed by a large-scale AI model to finally determine what is in the image. However, to do so, engineers need to collect a huge amount of data and use it to learn, and it is not realistic from the viewpoint of cost and human resources to prepare this for each project.
To accelerate the development of vision-based AI solutions in the future, we believe that it is necessary to remove these hurdles, which led to the development of Sony's unique edge AI sensing platform, AITRIOS.
AITRIOS and IMX500
Thank you. So, could you tell us what kind of service you offer about AITRIOS?
First, you need to know about the IMX500 intelligent vision sensor, which is a compatible device of AITRIOS.
The IMX500 is the world's first image sensor equipped with an AI processing function and has a stacked structure in which pixel chips and logic chips are superimposed, and the logic chip side is equipped with an AI-based image analysis processing function. As a result, necessary information about the object can be output as metadata (semantic information).
Since the amount of data can be significantly reduced compared to when sending images to the cloud, it is possible to reduce data transfer delay time when using cloud services, reduce power consumption and communication costs, and consider privacy.
The IMX500 is the world's first intelligent vision sensor equipped with AI processing functions.
The AI processing function is installed on the chip, and the information output from the sensor can be selected.
Created by PEAKS MEDIA by reference to materials provided by Sony Semiconductor Solutions Corporation.
The content has been translated by Sony Semiconductor Solutions Corporation.
When we talk about lightening image data, we generally imagine thinning out the data by lowering the resolution and sending it to the cloud, but does that mean that the image as it is captured is used for AI processing on the edge side?
To explain it in an easy-to-understand and concrete example, in vision sensing using a general image sensor, for example, if you want to know how many people are in the picture, you send the captured image data to the cloud, run the AI model on the cloud, and obtain the result that there are five human-like objects.
On the other hand, with the IMX500, AI processing can be performed on the image sensor side, so only the necessary information in the sensor (in this case, the count result of "5") can be retrieved and sent to the cloud. Instead of sending a large amount of image data to the network as in the past, only the necessary data can be sent, so it does not occupy network bandwidth.
Created by PEAKS MEDIA by reference to materials provided by Sony Semiconductor Solutions Corporation.
The content has been translated by Sony Semiconductor Solutions Corporation.
There is a clear difference in the amount of data to be transmitted between image data and simple text data. So, what role does AITRIOS play?
If only metadata is output from the sensor, it will have great advantages in terms of privacy and security considerations, and it is expected to be used in smart retail and smart cities. Developing these solutions requires collaboration with a number of partners. AITRIOS provides functions and development environments that partner companies can use to easily develop various applications using the IMX500.
By using AITRIOS, it is possible to manage devices such as replacing and updating AI models on the platform, so it is suitable for large-scale deployment in multiple places. Depending on the target application, we can flexibly respond on a case-by-case basis to how to identify the image and what kind of metadata to obtain as output. In addition, we provide one-stop tools and development environments necessary for development, such as samples of AI models for identifying attributes, and promote the development of solutions through co-creation.
Source: Sony Semiconductor Solutions Corporation
AITRIOS is also being used in the retail and logistics industries.
I understand that AITRIOS is being developed and demonstrated in various fields, but could you give us specific examples?
AITRIOS started its business at the end of 2021, and the response to this concept has been tremendous as a solution to problems related to the development of AI applications using images. People who saw the exhibition were amazed that such a small image sensor could do so much.
A good example of a use case in the retail industry is the Shelf Detection Solution. Demonstrations were conducted in line with actual operations, such as scanning shelves with an image sensor, detecting inventory shortages, and issuing instructions for replenishment, and many operating companies have praised the one-stop solution by AITRIOS for making it easy and with fewer resources to detect shelves.
As a specific example, Seven-Eleven has introduced the "AITRIOS-based Visual Detection Solution" to 500 stores in Japan with the aim of improving the advertising effectiveness of digital signage (electronic advertising).
This is a solution that automatically detects the customer's visibility directed at digital signage and grasps the number of viewers, viewership rate, etc. To measure the effectiveness of advertisements, there is a need to know the attributes of the people who listened to the audition, but in the past, consideration of privacy by using a camera was an issue. However, by using AITRIOS, only metadata is output and no image data that can identify individuals, so it is possible to measure the effect in consideration of privacy. These features were highly evaluated, and we decided to adopt them.
The IMX500-equipped camera is installed in a convenience store. Compact form factor of 55×40×35mm (excluding lens).
Caption courtesy of Sony Semiconductor Solutions Corporation
Image of shelf detection Real-time visualization of product shelf shortages and display disturbances.
It is also expected to contribute to the efficiency of sales floor management and labor savings. Source: Sony Semiconductor Solutions Corporation
Another example is the efficiency of the shop floor in the logistics industry.
In warehouse yards and other places where the internet infrastructure is vulnerable, it is difficult to transmit a large amount of vision data in real time, and it is difficult to grasp the situation at the site and visualize the waiting time for cargo and cargo handling work. By detecting and recognizing the license plate of a transport truck entering the yard using the IMX500 camera and AITRIOS, the system receives and records entry and exit information, obtains actual data on loading and unloading of cargo, which was difficult to manage in the past, and contributes to the reduction of unnecessary waiting time and work time for drivers through data analysis.
We are a platformer that provides AITRIOS, but the difficulty of vision sensing cannot be easily solved by simply developing and providing hardware/AI applications.
To actually implement a solution, it is necessary to have a deep understanding of the issues that exist across each industry, and at AITRIOS, a team specialized for each industry understands the actual situation and requirements that differ from application to application and supports the development and proposal of appropriate solutions. Sony has the technology and expertise it has cultivated in the field of image sensors. In this context, I feel that because I have accumulated experience in developing hardware for various industries, I can stand at the same level of understanding as partner companies that develop and provide solutions and have conversations to solve problems.
Sony's Challenge: Toward a Recurring Business Combining Hardware and Software
In what direction will AITRIOS develop in the future? Please tell us about the challenges of dissemination and what kind of impact it will have on society.
The areas in which the vision can play are important, but only partial. There are various sensors in the world, and there is a technology called "sensor fusion" that derives more accurate results by combining information obtained from multiple sensors. There are applications that make this possible, and we aim to accelerate DX through sensor fusion.
However, the technology and knowledge to create applications for each sensor are different, and the environment for realizing sensor fusion and the human resources who can use the technology are currently limited. I hope that AITRIOS can make a difference in this situation.
In the area of DX, we believe that there are things that can be created not only by supplementing the human eye with sensing, but also by combining various sensors. For example, by combining it with data that can be obtained from other sensors, it can contribute to labor saving and acceleration of unmanned operation. The IMX500 will continue to be an important product for AITRIOS, but we hope to be able to connect other Sony sensors to the AITRIOS platform in the future.
AITRIOS has a function to register AI models and sensing applications developed by partners and to download the ones you want to use, and you can implement the sensing applications obtained and developed in this way on the IMX500 via an AI converter. In addition, we plan to provide various SDKs so that the results of sensing applications built on AITRIOS can be easily linked to external systems.
We intend to contribute to the realization of sensor fusion by expanding the use of vision sensing with the IMX500/AITRIOS. To ensure that society recognizes the new value that sensor fusion brings, we would like to continue to accumulate successful experiences together with our partner companies.
Thank you. Finally, could you tell us about your thoughts on AITRIOS?
The AITRIOS business also represents a challenge for Sony. Until now, the image sensor business has been dominated by "selling goods" by selling hardware, and the market share of image sensors has remained the top in the industry. However, as a future business strategy, in order to further enhance the differentiation of our technological capabilities, we are aiming to combine our strengths of image sensors and software technology to create both a "experience business" in which we sell things as a business and a "recurring business" that generates continuous profits.
To achieve this, we need to make efforts to add value to our customers. We believe that activities that firmly grasp the characteristics and needs of each customer's industry and link feedback to service development will greatly contribute to the success of the AITRIOS business. Dialogue with stakeholders, including end users, is very important.
We aim to create a large number of use cases of AITRIOS in the future, and we are looking for a wide range of colleagues who can create value using AITRIOS. We regularly hold AITRIOS meetups to provide a place for people with ideas to actually experience AITRIOS. If you are interested, you will be able to experience the series of tasks from setting up the hardware, developing an AI model on the platform, deploying it to the hardware, and visualizing the detection results.
A meetup event at AITRIOS, which is run by Anna Maruyama. In addition to disseminating information online, we also operate a community that crosses industries
and occupations, centered on AITRIOS in real life. Source: Sony Semiconductor Solutions Corporation
PEAKSMEDIA EDITORIAL TEAM
PEAKS MEDIA is a website that disseminates news articles and useful information that supports the transformation of the manufacturing industry and open innovation.
We will introduce a variety of "PEAKS" such as outstanding technologies, materials, people, and companies in each project, and deliver information that will support innovators in companies who are promoting reform.