GPUImage – Overview

As a quite famous and professional framework for image processing in iOS, GPUImage has been released for a long time (the first version were published in 2013). In this post, I would like to share my own ideas and thoughts about it. And the cover is the one of works created by my first application with GPUImage.

When I first heard about GPUImage, it was in the autumn of 2015, because of a new application project. Before that, I did my first job as an iOS developer in quite tranditional IT company in Shanghai, which mainly provides services for commercial firms. So you can image that my daily work was so tedious, all kinds of uses of TableView and CollectionView, data request and UI layout adaption, that’s it. Until now I still feel not good for dealing with the layout in iPad when it changes its orientation, while there are some useful frameworks can help to address this kind of situation.

After that, I went to Hangzhou. And I always think that was one of the best choise for my life, even if at the beginning the work was so tough. There was only one guy doing coding stuff, yes that was me. Also, the product manager was always late and always liked to change his mind. For that application, a totally new prototype were released every month in average. And I spent couple days to read the source code of the previous versions to grasp the implementation of filters and stickers. CoreImage, a build-in framework can easily implement a simple filter function. And the sticker function are achieved by using normal View — UIView. Those kinds of methods for image processing were usually mentioned by interviewees I met, who thougth they already can handle most issues of image processing in mobile devices.

And someday the product manager pulled a requirement which was to implement a function like the layer in PS. That was a so incredible abstacle for me who only know how to requrest and update data at that time. Obviously, it costed me several days and it was luck that I found a class ‘CALayer’ in ‘CoreAnimation’ framework can do it perfectly. In addition, I was suprised by the functions of its sub-classes, which can deal with almost all kinds of animations and UI effects. And next, the biggest problem was that I had realised that only like some specific image processing frameworks can help me to develop an application with complicated image processing functions like OpenGLES, OpenCV or Metal, while I didn’t have enough time to learn how to use them. Lastly, I found GPUImage from GitHub, and it saved my job.

The wiki of GPUImage in GitHub is not a concrete manual. You only can get three points from it:

  1. GPUImage is based on OpenGLES 2.0;
  2. The performance of GPUImage is better than Coreimage;
  3. The process of adding a filter to a static image, a video or a frame captured by camera and displaying the result on GPUImageView or generate a file with a specifi format.

Maybe it’s because I only can understand these three points, so I still could not know some other things of its execution process, like: why the function ‘processImage‘ must be executed after the code of adding filters; it would crash if the function ‘useNextFrameForImageCapture‘ was not executed before export processing result as an image; and what is glsl used for? Moreover, I could not find more information from the Internet at that time no matter English or Chinese. Hence, I almost gave up to use GPUImage to implement those complicated image processing requirements.

Then, I wanna say some my roughly understanding of GPUImage. If there are some problems or mistakes found, pls correct me, cheers.

1. Fragment Shader and Vertex Shader

OpenGLES processes images through Fragment Shader and Vertex Shader. GPUImage is a encapsulated and extended framework based on an iOS build-in framework ‘GLKit‘ which is encapsulated from OpenGLES. And the version of OpenGLES could be selected in GLKit, while the OpenGLES version in GPUImage is OpenGLES2.0. We should know that there are various differences betweent different versions of OpengGLES. In my opinion, the most filter functions or classes provided by GPUImage are implemented by a series operations through Fragment Shader. This is because the results of the most filters do not change the size and shape of the processed image, and it’s more like to generate new pixels by calculating the original pixels. This is one of the most usual uses of glsl, which can define the process of pixel calculation methods. So glsl is very important part of GPUImage if you wanna use it to generate unique and amazing results.

2. Pipeline

GPUImage provides a concept of pipeline. Unlike the line in Masonry, GPUImage treats each input, filter and output as a piece of pipeline, and only those pieces of pipeline are connected in series, the image information could be transmitted through each independent piece of pipeline as input and finally get to result. So this is my own simple understanding of the execution process of GPUImage.

3. MVP

As I mentioned previously, GPUImage makes every independent piece of pipeline be connected in series by a quite special design pattern in iOS development, which is called MVP. The compulsory classes for the execution process, as well as the classes which are allowed to connected with others, have some common features:

  1. They are all the sub-classes of GPUImageOutput except GPUImageView;
  2. They all obey the protocol of GPUImageInput, except the input classes (GPUImage provides five kinds of input classes which are GPUImagePicture、GPUImageRawDataInput、GPUImageMovie、GPUImageUIElement、GPUImageVideoCamera. It’s easy to know what types of input these classes can deal with).

The class ‘GPUImageOutput‘ is not used directly during the whole process, because all output classes used are its sub-classes. It can be noticed from their names that the general use of GPUImageOutput is to be used as a output and the classes following the GPUImageInput protocol is used as an input. So during the process of connecting each piece of pipeline:

  1. The first object is only as an input, so it must be one of the five input classes and also not follow the GPUImageInput protocol;
  2. The classes which need to be connected in the middle of pipeline, like GPUImageFilter, as well as the father class of all filter classes, not only is the sub-class of GPUImageOutput but also obey the protocol of GPUImageInput. This is because it needs to reveive data from its privous node, and transmit the data to the next node after processing done;
  3. As the last node/object, GPUImageView is not necessary to be the sub-class of GPUImageOutput, in order to no next node at all.

Those are three key points of GPUImage I suppose. Actually I wrote this post in Chinese almost 2 years ago. You see, time is always so fast. And I’ll keep translating other posts about GPUImage in English and hope if there is some chances that I can modify GPUImage or develop a new image processing framework which could implement more interesting effects.



CGContextRef CGBitmapContextCreate (

void *data,

size_t width,

size_t height,

size_t bitsPerComponent,

size_t bytesPerRow,

CGColorSpaceRef colorspace,

CGBitmapInfo bitmapInfo



  • data 指向要渲染的绘制内存的地址。这个内存块的大小至少是(bytesPerRow*height)个字节。使用时可填NULL或unsigned char类型的指针。
  • width bitmap的宽度,单位为像素
  • height bitmap的高度,单位为像素
  • bitsPerComponent 内存中像素的每个组件的位数.例如,对于32位像素格式和RGB 颜色空间,你应该将这个值设为8。
  • bytesPerRow bitmap的每一行在内存所占的比特数,一个像素一个byte。
  • colorspace bitmap上下文使用的颜色空间。
  • bitmapInfo 指定bitmap是否包含alpha通道,像素中alpha通道的相对位置,像素组件是整形还是浮点型等信息的字符串。



const CGSize size = size;
const size_t bitsPerComponent = 8;
const size_t bytesPerRow = size.width * 4; 
CGBitmapContextCreate(calloc(sizeof(unsigned char), bytesPerRow * size.height), size.width, size.height, bitsPerComponent, bytesPerRow, CGColorSpaceCreateDeviceRGB(), kCGImageAlphaPremultipliedLast);

2.only alpha

const CGSize size              = size;
const size_t bitsPerComponent  = 8;
const size_t bytesPerRow       = size.width; 
CGContextRef context = CGBitmapContextCreate(calloc(sizeof(unsigned char), bytesPerRow * size.height), size.width, size.height, bitsPerComponent, bytesPerRow, NULL, kCGImageAlphaOnly);



UIButton * btn0 = [[UIButton alloc]initWithFrame:CGRectMake(100, 450, 50, 100)];
[self.view addSubview:btn0];
UIButton * btn1 = [[UIButton alloc]initWithFrame:CGRectMake(200, 450, 50, 100)];
[self.view addSubview:btn1];
[[btn0 rac_signalForControlEvents:UIControlEventTouchUpInside] subscribeNext:^(UIButton * sender) {

//do something...

btn0.selected = YES;

btn1.selected = NO;

[[btn1 rac_signalForControlEvents:UIControlEventTouchUpInside] subscribeNext:^(UIButton * sender) {

//do something...

btn0.selected = NO;

btn1.selected = YES;



但是在以上代码中,我并没有创建全局属性并且也没有在Block中调用任何全局属性,结果同样出现了循环引用。我已经意识到了问题肯定出在Block中,就顺着代码排查可能性。我先把btn1的事件响应信号的接收Block中的btn0.selected = NO;去掉,发现问题解决了。因此意识到,即使是局部变量对象,如果存在同样的情况,也会导致循环引用。



__weak UIButton * weakBtn0 = btn0;
__weak UIButton * weakBtn1 = btn1;




Method originalDealloc = class_getInstanceMethod(self, NSSelectorFromString(@"dealloc"));

Method swizzledDealloc = class_getInstanceMethod(self, @selector(ac_dealloc));

method_exchangeImplementations(originalDealloc, swizzledDealloc);
- (void)ac_dealloc {

[self ac_dealloc];

NSLog(@"buffer dealloc:%@",self);



更正:交换方法中添加的自定义操作一定要在[self ac_dealloc]之前。

- (void)ac_dealloc {

//do something

NSLog(@"buffer dealloc:%@",self);

[self ac_dealloc];






if ([super respondsToSelector:@selector(methodName)]) {

[super performSelector:@selector(methodName)];



Method oriM = class_getInstanceMethod([super class], @selector(methodName));

SEL selector = method_getName(oriM);

[super respondsToSelector:selector];

如会出现“undeclared selector”的warning时


#pragma GCC diagnostic ignored "-Wundeclared-selector"


#pragmaclang diagnostic push
#pragmaclang diagnostic ignored"-Warc-performSelector-leaks"

warning code...

#pragmaclang diagnostic pop

UICollectionView prefetch


if ([_inputSelectView respondsToSelector:@selector(setPrefetchingEnabled:)])
    _inputSelectView.prefetchingEnabled  = NO;


@interface NLViewController <UICollectionViewDelegate, UICollectionViewDataSource, UICollectionViewDelegateFlowLayout, UICollectionViewDataSourcePrefetching>


static NSString *cellIdentifier = @"cell";

@implementation NLViewController

#pragma mark - View lifeCycle

- (void)viewDidLoad {
[super viewDidLoad];

collectionView.delegate                      = self;
collectionView.dataSource                    = self;
collectionView.prefetchDataSource            = self;


- (void)tableView:(UITableView *)tableViewprefetchRowsAtIndexPaths:(NSArray*)indexPaths {    

for (NSIndexPath *indexPath in indexPaths) {




从一定角度来看,collection view 的预加载请求只是试图优化未来不确定状态的一种猜测,这种状态可能并不会真实发生。例如,如果滚动速度放缓或者完全反转方向,那些已经请求过的预加载 cell 可能永远都不会确切地显示。

- (void)tableView:(UITableView *)tableViewcancelPrefetchingForRowsAtIndexPaths:(NSArray *)indexPaths;

NSNotification object


  • post填nil, add填XXX。此时收不到通知。
  • post填XXX, add填nil。此时收到通知。
  • post填XXX, add填XXX。此时收到通知。
  • post填nil, add填nil。此时收到通知。
[[NSNotificationCenter defaultCenter] addObserver:self selector:@selector(XXXX:) name:@"XXXXX" object:objectA];
[[NSNotificationCenter defaultCenter] postNotificationName:@"XXXXX" object:objectA userInfo:@{@"key":@"param"}];


The object whose notifications the observer wants to receive; that is, only notifications sent by this sender are delivered to the observer. When nil, the notification center doesn’t use a notification’s sender to decide whether to deliver it to the observer.




  1. CALayer包含在QuartzCore框架中,这是一个跨平台的框架,既可以用在iOS中又可以用在Mac OS X中。

2. 在iOS中CALayer的设计主要是了为了内容展示和动画操作,CALayer本身并不包含在UIKit中,它不能响应事件。所以在使用过程中,需要通过手势来控制CAlayer的属性变化(transform啦等等)时需要用一个view来接收手势(touch事件或者GuestureRecognizer),并在事件响应方法中实现CAlayer的变化。

3. 由于CALayer在设计之初就考虑它的动画操作功能,CALayer很多属性在修改时都能形成动画效果,这种属性称为“隐式动画属性”。但是对于UIView的根图层(view.layer)而言属性的修改并不形成动画效果,因为很多情况下根图层更多的充当容器的做用,如果它的属性变动形成动画效果会直接影响子图层。

4. UIView负责创建并管理CALayer。比如一个:viewA上add了一个viewB,则viewB.layer也成为了viewA.layer的sublayer。

5. CALayer的各种属性:

  • contents id类型,就是啥类型都能对他赋值都不会报错,但是赋值完看上去有效果的只有图片类型。IOS开发中具体实现为:view.layer.contents = (id)[UIImage imageNamed:”XXX”].CGImage; 听说MAC开发中需要用NSImage。UIImageView内部就使用这个东西实现图片显示。
  • contentGravity = view.contentMode
  • contentsScale = view.contentScaleFactor
  • maskToBounds = view.clipToBounds
  • contentsRect 跟contents有关系的属性,就是规定了contents的可视区域。但是通过这个设置的可视区域只能为一个矩形区域。
  • contentsCenter CGRect类型,规定了contents的放大区域,当设置contentsScale时这个区域就变大啦,以外区域就缩小啦

6. (这段完全是载的)CGImage并不是唯一可以赋值给contents属性的,也可以使用Core Graphics绘制寄宿图给它,如果你实现了drawRect方法,然后如果你调用setNeedsDisplay或者外观属性被改变时,它就会自动调用drawRect自动重绘,虽然drawRet是一个UIView方法,但是其实都是底层都是CALayer重绘保存了图片,如果你不需要自定义绘制就不要写一个空的drawRect方法,它很消耗cpu和内存资源,CALayer有一个可选的delegate属性,如果设置了delegate,并主动调用了layer的displey方法(注意和drawRect不同这个重绘时机是开发者自己控制的,也可以调用setNeedsDisplay方法给系统自己找时机调用),它会调用displayLayer方法,在这里是设置contents属性的最后机会了,如果你没有实现这个方法,它会尝试去调用下面这个方法:drawLayer。。。,如果你实现了displayLayer方法,drawLayer就不会调用了

7. 其实!UIView的frame、bounds、center、transform这些属性只是存取方法!真正实现UIView显示出来的样子的都是通过这个控件的layer来实现。所以 = view.layer.position 


view.frame = view.layer.frame    


view.bounds = view.layer.bounds
view.transform = view.layer.CGAffineTransform 



  • geometryFlipped 这个用来翻转layer所在的坐标系,上下翻!就是!y轴负的跟正的翻!
  • zPosition 看名字就能明白是用来设置layer的层级位置的,默认为0,越大越往上!

8. 关于layer的自适应: