Apple patent is for multimedia scripting
Posted by Dennis Sellers 
Jan 17, 2008 at 11:37am
An Apple patent (20080016491) for multimedia scripting has appeared at the U.S. Patent & Trademark Office. Disclosed are methods and systems for multimedia scripting, including evaluating a script at runtime and invoking a process for editing multimedia in dependence upon the script.
According to Apple, multimedia may include a still image and video images. Multimedia scripting may also include accepting text entered into a text-input graphical user interface as a script for runtime evaluation, accepting from a non-text-based graphical user interface a designation of scripts for runtime evaluation, and effecting a disposition of the edited multimedia in dependence upon a script, such as storing the multimedia as a file, presenting the multimedia, or encoding the edited multimedia as an email attachment.
Here’s Apple’s summary of the invention: “Among other advances, the invention seeks to solve the problems and meet the needs and desires described above. In doing so, some embodiments of the invention include methods, systems, and computer program products for multimedia scripting, namely editing digital multimedia files and digital multimedia presentation using a scripting language. In preferred embodiments, the scripting language is human-readable, which facilitates scripting by non-programmers. Mulitmedia may include audio, video images, or still images or operations thereon. In some embodiments, multimedia scripting may also include effecting a disposition of an image or other multimedia, such as, for example, to a file system or to an email attachment.
“In a more specific embodiment of this type, a high-level program interface includes graphics filtering functions for editing digital images that are capable of being invoked by the use of a scripting language. In some embodiments, the user exploits a high-level program interface by creating effects or specifying filter functions from a pre-defined list by using the scripting language. The scripting language may be capable of adding filters to the pre-defined list. Scripting may also be used to make a Core Image graph, as described in more detail below. Alternatively, a Core Image graph may be parsed to define a user-level script which could be used to recreate the Core Image graph. Other embodiments include a graphical user interface (GUI’) for script creation. This GUI may be adapted for user-level script creation or for developer script creation.
“In one embodiment of the invention, a human-readable scripting language, upon being interpreted, will invoke software exploiting a selected processor in the system to compose a graph-like description of an image task. Invocation of this software may be carried out by calling from an interface to external native modules written in a traditional compiled language. Invocation of the software may alternatively be carried out by sending a message from an interface to external native modules written in a traditional compiled language. Core Image may include an API and an efficient processing infrastructure for deployment in a multi-processor system. The graph-like description may be a node and link representation of an image, where nodes may represent operators and links may represent intermediary results and the storage necessary to hold those results. In greater particularity, nodes in the graph-like description may ultimately comprise threads or programs for computing a portion of the overall image operation on another processor. Furthermore, having a graph-like description of the overall image task allows use of an optimizing compiler to reduce the necessary resources for the overall image task. This compiling function is especially useful since the node programs will generally run on a processor other than that which runs the compiler.
“The scripting language used to invoke image processing software such as Core Image and audio processing software such as Core Audio is referred to in this specification as the Core Image Scripting Language (‘CISL’). The term ‘CISL’ is also used in this specification to refer to the scripts written in such a scripting language, the software interpreting such a scripting language and invoking Core Image and Core Audio as described immediately above, and any interface adapted for CISL instructions. Although Core Image and Core Audio are used as examples, CISL may be used with other image processing software or audio processing software, and is not confined to these specific examples, which are provided merely for illustration. CISL software may be contained in one or more CISL modules separate from Core Image and Core Audio, or it may be embedded in a Core Image or Core Audio module. Differing embodiments of the invention may exploit either or both of the language or the software as a definition of CISL. The specific meaning derives easily from the contextual use.
“The forgoing general embodiment may be described in the context of a contemporary pairing of a single CPU with a single GPU. This embodiment proposes software running on the CPU for providing a user-iterface, assessing the overall image task, and constructing a graph-like description of this image task. This may be visually represented as a tree graph of nodes and links with associations as described above. Since the node-programs may execute on the GPU, the construction of the program accounts for properties of the GPU. Most notably, and in a general sense, programmable GPUs operate several parallel execution streams so the node-programs may be expressed in parallelizable languages. For example, node programs may be GPU fragment programs. After construction of the graph representing the overall image task, the graph may be optimized by virtue of a compiler running on the CPU. Alternatively, the graph may be optimized by a compiler in distinct pieces, as the graph is created. The purpose of optimizing is to minimize memory usage and CPU or GPU time or otherwise gain efficiency when the image is computed.
“According to varying embodiments of the invention, optimization may have many functional characteristics. For example, optimization may include caching intermediary results, consolidating multiple fragment programs into one, limiting memory and computation to areas within a confined domain of definition and region of interest, or optimizing division of computation between processors.
“Applying these techniques in the contemporary graphics context is highly efficient and allows developers to write filters by expressing the operations to be performed on an element (e.g. pixel) or elements without concern for the specific hardware in a system—that will be accounted by the compiler. In addition, having created a scripting language, an API, and efficient processing infrastructure for deployment in a multi-processor system, many embodiments also include functionality to exploit the scripting language on single processor systems. In a very general sense, this is accomplished by emulation.”
The inventor is Frank Doepke. The graphic below is a sample hardware configuration.
The “Macsimum MWSF 2007 Coupon Book” is available for download. You can find it here and print it as a PDF. It has discounts, special offers and promotions in conjunction with the upcoming Macworld Conference & Expo in San Francisco.
Article Information
Comment on this Article Print this Article Email this Article Digg This
Contributor
Dennis Sellers
Dennis has been a newspaper editor/reporter (seven years) and teacher (seven years). He has over 4,000 magazine, newspaper and online articles to his credit. He has also covered the Mac and tech industries for over a decade for such online publications as MacCentral, MacMinute and now MacsimumNews.
