One of the most bright prospects being the impact of Artificial Intelligence on live streaming services. Live streaming is an on-demand digital media, videos, or audios which are displayed simultaneously while the provider is recording it. Live streaming has had its impact as early as the 90s with an increase in internet radio, webcast, and private channels. But the major outburst was created with the introduction of YouTube, where people realized that images can still reach a large number of people and broadcast the media. Later, many technologies, software, and apps came for Android, iOS, and PC for live streaming. Examples are end-to-end technology, flash-based or Java plugins, Windows Media or Real Media, and streaming over public address systems. The effect these have made is still a better way to connect with the audience and create lots of fanbase. The best example today is PewDiePie, a successful YouTuber who got most of his subscribers by recording live streams of his gameplay and providing a comedy commentary. In the past, recording the game, converting it to video, and uploading it to YouTube was an effective but tough way to take the audience to the media and thus was not preferred by content creators. Today, he stands strong with 46 million subscribers, which can be proof of how live streaming can be the best way to connect with the audience.
From the age of the Industrial Revolution, the inception of technology has revolutionized the way we used to live, think, work, and act. Contradictorily, technology has a solution for a problem it has created, affecting in a positive way by simplifying work, reducing loads, making life easier, and the best part being, helping out people at their best. One of the major rises in technology of the 21st century is Artificial Intelligence. Artificial Intelligence is a method of making a computer, a computer-controlled robot, or software think intelligently, in the similar manner that intelligent humans think. It is a set of instructions or a way to create a computer to do a certain task. This can be a smarter way to organize a spreadsheet, to generate the output to certain meaningful data, or to solve a complex calculation. Artificial Intelligence has been in works since Stanford University first introduced the concept with the name of “Logic Theorist” in 1955, and Artificial Intelligence has surmounted with its techniques and provided us with its impact on various concepts of science and human civilization.
Overview of Artificial Intelligence
Machine learning, which is an evolved form of AI, is a process that allows a system to learn from data, improve, and make decisions based on said data without human programming. A simple example of machine learning in live streaming would be a recommendation system for an end user to efficiently find related content. The machine would learn from the end user and provide better results over time. High bandwidth and good CPU are required for the system to run efficiently. Machine learning has strong potential within the future of live streaming, but it would come with a cost for better results.
With this in mind, AI in regards to live streaming could be applied to mean the automation of a certain segment, let’s say a replay for instance. Digital replay systems have already been in place for a long time now and are used widely in broadcasts to show a critical play or sequence of events. Now the AI aspect would come from the system recognizing these critical points in the game and then finding and queuing the clips to replay. This would align closer to the general AI definition as these systems are emulating human decisions.
AI can be categorized as either applied or general. Applied AI is far more common and is at the root of the majority of AI applications alive today. It is simply a program that is designed to do something a human would normally do. General AI is more concerned with automation of the human thought process.
Introduction to Live Streaming Services
Live streaming services started a new era of digital transformation in TV services. With the inception of live streaming services, traditional TV started losing its irrefutable importance in terms of entertainment because live streaming services were promising where, when, and how the viewer wants. It is because live streaming services rely on the internet and cloud computing. According to Statista, live streaming video captures 17% of the online video market across devices and platforms. In the initial stage of live streaming services, there were many drawbacks in terms of quality, lag, smoothness, management cost, and storage cost. But in the start of 2010, the entire scenario started changing, and constantly studying Artificial Intelligence (AI) is one of them that is impacting live streaming services. High efficiency in terms of quality in audio and video, smoothness, reduced cost, and reduced human efforts in managing the stuff are the main focus of AI in live streaming services. AI has many areas to focus on in live streaming services. Among them, automation, cloud computing, content management, serverless computing, search and recommendation, transcoding, image detection, analysis, recognition, machine learning, audio and video analysis are the main areas that AI is focusing on to make the live streaming services more reliable and cheaper.
Benefits of Artificial Intelligence in Live Streaming
In the case of AI, this value comes in the form of personalized viewing lists. By assessing the user’s data and behavior patterns, AI will be able to suggest content tailored to the individual, rather than the viewing populace as a whole. This could extend beyond preference in genre and actors, to the extent of predicting how tolerable an individual is to ambiguous movie endings. For the severe cases, AI could even filter out mature content where inappropriate for children. This capability could decrease user decision time dramatically and increase satisfaction with content. A win for the user and the streaming service.
In an industry where user engagement is the ticket to growth and retention, content is king. However, in the ever-expanding globalized pool of available content, users are frequently overwhelmed with choice, leading to decision fatigue. Content recommendation has long been the solution to this problem, from simple genre-based lists to more complex recommendation engines similar to that of Netflix that rely heavily on behavioral data. Artificial Intelligence will have a huge impact on this area, due to its ability to sift through and interpret massive amounts of data, more accurately than current methods and with lower costs. In fact, it’s reported that 57% of consumers are okay with providing personal data to companies, as long as the tradeoff is something of value.
Personalized Content Recommendations
The benefits of achieving this kind of system are twofold. Firstly, the provider can be safe in the knowledge that their service is more accessible to the viewer and offering them a chance to view the full extent of the content on offer. This can be seen as essential for video on demand services which rely on keeping the viewer on their site as opposed to a competitor’s. A study in 2017 found that the average viewer spends 90 seconds looking for something to watch on Netflix before going elsewhere. This is likely due to a lack of immediately enticing content on the homepage and the fact that the viewer may not know exactly what they are looking for.
Typically, live streaming services have always provided users with an extensive choice of content channels and options. This, in itself, can be seen as a daunting mass of information to navigate through when all the viewer really wants is the right content at the right time. Imagine being greeted by a TV guide that only has one channel, with knowledge of everything that is on and the ability to flick between this channel and tailored alternatives at any given time. This all-encompassing intelligent media guide is the underlying goal for personalized content recommendations in live streaming.
Personalized content recommendations aim to utilize an in-depth understanding of each viewer in order to present them with the most relevant content. This is achieved through utilizing data about the user and their context to provide them with a personally optimized channel of content. This concept is the core of consumer AI and spans a wide array of potential applications.
Enhanced Video Quality and Delivery
The idea behind these different types of coding is to reduce the bandwidth required to transmit and store video data and to minimize loss of quality.
Another process used for video coding is the motion compensation and block-based prediction. This process takes each frame and separates it into blocks of pixels. Any subsequent frames will exhibit block movement. This can be determined using a block matching algorithm to determine the location of the block in a subsequent frame. Blocks of pixels are compared for their similarity in each frame, and block motion can then be extrapolated. Data on movement is transmitted with the absence of the moved block.
There are two types of video coding. Discrete Cosine Transform coding maps blocks of pixels into the frequency domain using a Fourier method similar to audio coding. DCT is one of the most important techniques in the field of Image and Video Compression in recent years. DCT produces a two-dimensional spectral scan of a two-dimensional image, and the conversion applied by the DCT is completely reversible.
Quality is the core of any video material. Overall quality of video and audio, clarity, smoothness in viewing, colors, and screen content are of primary importance. Generally, higher quality is desirable, but must be balanced against the increased cost of bandwidth and storage media. It is understood that video quality is compromised when received over a low bandwidth link.
Real-time Audience Engagement
Comment moderation is a notable issue for channels that reach a large audience. While the streamer wants to provide an open platform for discussion, abusive, racist, sexist, and other “toxic” comments can all be highly damaging towards the streamer’s image, as well as towards the experience of other viewers. Moderator bots have been a common tool utilized in the past but are often ineffective for their impact vs the amount of work required to maintain them. Ecan has developed a new tool on their AI Purity that allows the streamer to set a tolerance level for abusive content, with a filter that will automatically hide any comments beyond the set level, and a learning algorithm that will improve the filter’s accuracy based on the streamer’s decisions. This can potentially provide an improvement over the moderator bots as there is no need to assign anyone to the task, and it will improve over time as long as the streamer continues to use it.
The drive to engage more viewers and drive interaction in the live arena has led to the creation of many AI-powered real-time audience engagement tools. These tools are designed to keep the audience tuned in throughout the entirety of a live stream. Implementing some of these new systems could require a lot of effort and potentially a high cost, but it could also lead to a much more engaging experience for the viewer, which is valuable in its own right.
Challenges and Limitations of Artificial Intelligence in Live Streaming
An interesting and perhaps less considered issue is the impact an increased use of AI could have on employment in the live streaming industry. The potential benefits of employing AI to automate many tasks or create entire streams could result in job losses in some areas. While it can be argued that AI will open new job opportunities in its development and maintenance, this may not be of any benefit to those who have lost their jobs, and newly created jobs will likely require a different skill set.
Privacy and data security concerns are significant issues for any company working with AI, and this extends to live streaming services. Using AI to potentially monitor and record a user’s viewing habits to tailor content specifically for them could contravene data protection laws. There is then the issue of data breaches, which become even more costly if large databases of personal information are stored to be used in machine learning algorithms. If the potential benefits become outweighed by the legal and financial liabilities, then AI will not be worthwhile for live streaming services.
This section focuses on the challenges and limitations of AI in live streaming. These are critical issues to address as AI use becomes more widespread in the industry over the next decade. A balance needs to be struck between the potential benefits of AI and its limitations, and any negative impacts need to be negated. Failure to do so would result in the potential benefits not being realized or not being worth the cost.
Privacy and Data Security Concerns
This is still a time of uncertainty for AI implementations, as strict specifications on what methods of data processing are acceptable have not been set and have to be determined on a case-by-case basis, and avoiding risk is likely to be prioritized over potential gains. This could either prevent any AI implementations to avoid controversy and the need for custom legal advice, or it could spawn a new sector of legal consultancy to specifically translate the regulations for AI.
On the 25th May 2018, the General Data Protection Regulation comes into effect, requiring all organizations, including live streaming services, to require an affirmative opt-in for the processing of personal data and also allowing user access to all tracked data about them. This is a positive step towards giving users more control and knowledge about their own data, and it applies to all new or traditional methods of data processing so should cover any AI methods.
This was shown recently when the Electronic Privacy Information Center (EPIC) released a report documenting how the US Federal Trade Commission had failed to enforce the 2011 consent order against Facebook and its plans to use biometric facial recognition data. EPIC had discovered Facebook was using this data to improve its AI algorithms, clearly a move that goes against the order; but the FTC took no action to stop this and merely halted the facial recognition for new users and deletion of data for tagged photos for other users. This incident undermines the public’s future trust in AI and the security of their personal information, from a popular site used regularly for socializing, to AI used in any other sector including live streaming.
Regardless of the technological advancements and benefits that AI can provide to live streaming and content curation, there are serious concerns about the privacy of user data, the security of the algorithms used to sift through it, and the potential for severe data breaches. The legal guidelines on data protection are still being extended to cover AI and may be too slow with this traditionally slow process.
Ethical Implications of AI-Powered Content Curation
The method of content curation, particularly real-time curation, presents several distinctive ethical concerns. One of the foremost problems is that of implied endorsement. When a piece of content is chosen for display, the content curator is essentially endorsing that content by giving it increased visibility. In cases where the content in question is objectionable, this could result in public relations fallout for the curator. Similarly, the stream hosts could be held accountable for the actions of third-party content providers. If an advertisement or content item is found to be illegal or objectionable, the host could theoretically be held liable for promoting such content. This would be particularly problematic in cases where the host is unaware of the nature of promoted content as a result of black box AI decision making. An extension of this problem is the potential for AI to make decisions based on biases of the developers or training data. An AI developed by a politically conservative organization might avoid promotion of content created by liberal artists and vice versa. With the increasing prevalence of AI curation, this could lead to content providers avoiding certain hosting services based on the anticipated action of the AI. This scenario results in a balkanization of the internet and could potentially fuel culture wars.
Technical Limitations and Reliability Issues
Measures can and should be taken to mitigate these issues, such as reducing the amount of dynamic data in content generation and using smarter AI implementations. However, it is a long road ahead until such systems can reliably serve their purpose in the live streaming industry.
Another issue is that the AI systems are often not reliable and do not have fail-safes or provide feedback to the user. This is particularly pertinent in content recommendation systems. If an AI system recommends content and the user dislikes it, there is no guarantee that there will be alternative content to suit the user. If the content is generated dynamically by an AI system, such as in the case of automated game commentary generation, a fault in the AI system can result in no content being generated at all.
AI systems rely on pattern recognition, which can break down in an environment that features a large amount of dynamic data and content that is ever-changing. If the data being processed is transformed in some way, the AI system may not be able to recognize it, leading to erroneous results.
The complex nature of AI systems and the wide supply of real-time data in today’s day and age can create a situation where an AI system consumes too much data and is unable to handle more data or processes. This can lead to bottlenecks and slow processing of data, which can have a severe impact on the streaming quality and the user experience.
Artificial intelligence has a long way to go before it becomes a ubiquitous force in the live streaming industry. One major issue is the technical limitations of AI systems. While chatbots and content recommendation systems are becoming more advanced, many AI systems don’t have the capability or the power to adapt to the large amount of unpredictability often found in live streaming services.
Future Trends and Opportunities in AI-Enabled Live Streaming
An AI-driven automated tagging system could take information given by the content creator and apply a detailed set of tags to the video without the need for manual input. High-level annotation of multimedia data has various uses for content creators, and due to the hands-off nature of automated tagging, it is a cost-effective technology. The easier the video is to be found, the larger the audience who may view it. Therefore, this tool would be a key asset in video marketing.
Video understanding and retrieval is a current research goal at the intersection of computer vision and multimedia, and progress in this area will surely provide benefits to both video creators and their audiences. Steps of progress would involve a possible increase in the amounts of automatically detected game and stream categories through analysis of the audio and video content, to the development of tools allowing streamers to categorize their video highlights for easier access by viewers.
In consideration of user-generated content, improved automated content tagging and advanced content creation both lead to increases in quality. As stated by representatives from Twitch, interactive entertainment is not just entertainment. Steps are beginning to be taken to provide viewers with the ability to search for and locate video of a certain genre or even a certain game from previous live stream events. This can be considered an information retrieval task. It involves searching through a large collection of video content to find material which is most relevant to the user’s query or criteria.
The technology and methods described in Section 3 all contribute to a value chain leading to the creation of live streaming content. The intended value of the content is to be perceived by an audience. The greater the quality of the content which the creator is able to produce, the larger an audience which may be attracted and retained, leading to increased monetization for the creator. This, in turn, creates an incentive for the creator to continue production of content.
Advanced Content Analysis and Tagging
Automatic language translation is also a growing area with the increase in available multimedia content in languages other than English. Given the source language and a text description of a translation task, machine translation systems can utilize the AV content to generate a multimodal translation that is presented as a translated version of the original content.
High-level content understanding goes beyond feature extraction and tries to automatically identify the events and concepts that occur in the media and relate them to predefined concepts or ontologies. This can enable automatic metadata generation to describe the media in a semantic way and also enable event-based navigation and visualization of media collections.
Content analysis and understanding are key to enabling content navigation, retrieval, recommendation, and visualization, which are the stepping stones of content-centric AI. Metadata formats for streaming content vary greatly, ranging from ID3 tags for MP3 audio to complex XML-based formats. AI techniques for content understanding make use of audiovisual (AV) features and text. They use audio and video analysis to extract AV features and possibly generate a textual description. The features are then mapped to metadata using machine learning techniques. With well-defined content analysis and metadata mapping, AI can enable very flexible content retrieval and recommendation using complex queries on the multimodal features and content similarity. This is an area that is well understood and is the basis of the many web-scale multimedia search and recommendation systems that exist today.
AI-Driven Content Creation and Editing
To respond to the rapidly growing demand for video and keep production costs to a minimum, Goldman Electronic Group launched the first search engine-style video database that connects video producers with a global market. The role of AI in this particular database is to automate the tagging and cataloging of video through the extraction of metadata, since nearly 80% of the world’s video is untagged. This is achieved through effectively watching the cataloging of what the video contains, i.e. people, places, dialogue, on-screen text, brand logos, colors, or any other information that could be used to understand and index it. Automated scene detection is used to segment videos into coherent scenes and apply metadata to an entire scene. This can be especially useful to an editor or producer looking for B-roll footage, since it can be used to produce a reel of all footage containing a specific subject. The decision to employ AI technology in video indexing comes as a result of the massive growth of the Internet, which particularly in areas such as IPTV, VOD, Mobile, and web video has made video content of all kinds more accessible and prevalent than ever. This can often mean that the video a person seeks is somewhere in a sea of other videos. While traditional search engines use text indexing to bring this content to the viewer, AI indexing allows the content itself to be indexed and even searched for, with the system retrieving the closest matching results to the query. This technique has potential applications outside the realm of professional video. With the rise of IoT and multimedia integration, it is feasible that one day even personal video collections could be indexed in this way.
Integration of Virtual Reality and Augmented Reality
Mixed reality environments involving VR and AR represent a large departure from the traditional content of live streaming media. This content of real and virtual interaction is characterized by a deeply complex ontology and a highly interconnected web of context. A universal example of mixed reality interaction is that of a chess match between a human and a virtual opponent. Modern AI does not possess the capability to understand and properly react or adapt to the myriad possible moves and strategies within a chess game. However, AI has come a long way in achieving this with respect to specific game titles, such as the ability to script a bot to play a custom map in Starcraft with very high proficiency. In transitioning such AI from standalone game environments to mixed reality live streaming content, we are again led to the issue of AI content generation and specifically AI-controlled virtual actors. Automatic capture of game dynamics and AI content analysis are likely to be employed in the translation of game-specific AI to virtual actors interacting within game-themed virtual environments. This is a very technical area that holds great potential for future research.
Also, live streaming services are likely to significantly leverage AI tools in integrating VR and AR technologies. The ultimate manifestation of VR/AR integration involves the creation of mixed reality forums within live streaming platforms using game engines such as Unity 3D, where users and AI-controlled virtual actors interact in AI-generated virtual environments to discuss real-world topics. While potentially groundbreaking, this level of VR/AR integration is unlikely to happen within the next decade. More feasible is the incorporation of VR/AR into live streaming eSports coverage, most notably of the games Dota 2 and Counter Strike, both of which have strong modding communities. In such cases, AI can facilitate automated camera control and shoutcasting AI-controlled virtual players. Analysis of gameplay and automatic generation of in-game cutscenes using AI are other possibilities. The sparse nature of current VR/AR content relative to the massive user base of gamer consumers represents a large market opportunity for developers of VR/AR technologies. AI-driven content generation and analysis, as previously discussed, will be instrumental in realizing this opportunity.
