GPUImage – Overview

As a quite famous and professional framework for image processing in iOS, GPUImage has been released for a long time (the first version were published in 2013). In this post, I would like to share my own ideas and thoughts about it. And the cover is the one of works created by my first application with GPUImage.

When I first heard about GPUImage, it was in the autumn of 2015, because of a new application project. Before that, I did my first job as an iOS developer in quite tranditional IT company in Shanghai, which mainly provides services for commercial firms. So you can image that my daily work was so tedious, all kinds of uses of TableView and CollectionView, data request and UI layout adaption, that’s it. Until now I still feel not good for dealing with the layout in iPad when it changes its orientation, while there are some useful frameworks can help to address this kind of situation.

After that, I went to Hangzhou. And I always think that was one of the best choise for my life, even if at the beginning the work was so tough. There was only one guy doing coding stuff, yes that was me. Also, the product manager was always late and always liked to change his mind. For that application, a totally new prototype were released every month in average. And I spent couple days to read the source code of the previous versions to grasp the implementation of filters and stickers. CoreImage, a build-in framework can easily implement a simple filter function. And the sticker function are achieved by using normal View — UIView. Those kinds of methods for image processing were usually mentioned by interviewees I met, who thougth they already can handle most issues of image processing in mobile devices.

And someday the product manager pulled a requirement which was to implement a function like the layer in PS. That was a so incredible abstacle for me who only know how to requrest and update data at that time. Obviously, it costed me several days and it was luck that I found a class ‘CALayer’ in ‘CoreAnimation’ framework can do it perfectly. In addition, I was suprised by the functions of its sub-classes, which can deal with almost all kinds of animations and UI effects. And next, the biggest problem was that I had realised that only like some specific image processing frameworks can help me to develop an application with complicated image processing functions like OpenGLES, OpenCV or Metal, while I didn’t have enough time to learn how to use them. Lastly, I found GPUImage from GitHub, and it saved my job.

The wiki of GPUImage in GitHub is not a concrete manual. You only can get three points from it:

  1. GPUImage is based on OpenGLES 2.0;
  2. The performance of GPUImage is better than Coreimage;
  3. The process of adding a filter to a static image, a video or a frame captured by camera and displaying the result on GPUImageView or generate a file with a specifi format.

Maybe it’s because I only can understand these three points, so I still could not know some other things of its execution process, like: why the function ‘processImage‘ must be executed after the code of adding filters; it would crash if the function ‘useNextFrameForImageCapture‘ was not executed before export processing result as an image; and what is glsl used for? Moreover, I could not find more information from the Internet at that time no matter English or Chinese. Hence, I almost gave up to use GPUImage to implement those complicated image processing requirements.

Then, I wanna say some my roughly understanding of GPUImage. If there are some problems or mistakes found, pls correct me, cheers.

1. Fragment Shader and Vertex Shader

OpenGLES processes images through Fragment Shader and Vertex Shader. GPUImage is a encapsulated and extended framework based on an iOS build-in framework ‘GLKit‘ which is encapsulated from OpenGLES. And the version of OpenGLES could be selected in GLKit, while the OpenGLES version in GPUImage is OpenGLES2.0. We should know that there are various differences betweent different versions of OpengGLES. In my opinion, the most filter functions or classes provided by GPUImage are implemented by a series operations through Fragment Shader. This is because the results of the most filters do not change the size and shape of the processed image, and it’s more like to generate new pixels by calculating the original pixels. This is one of the most usual uses of glsl, which can define the process of pixel calculation methods. So glsl is very important part of GPUImage if you wanna use it to generate unique and amazing results.

2. Pipeline

GPUImage provides a concept of pipeline. Unlike the line in Masonry, GPUImage treats each input, filter and output as a piece of pipeline, and only those pieces of pipeline are connected in series, the image information could be transmitted through each independent piece of pipeline as input and finally get to result. So this is my own simple understanding of the execution process of GPUImage.

3. MVP

As I mentioned previously, GPUImage makes every independent piece of pipeline be connected in series by a quite special design pattern in iOS development, which is called MVP. The compulsory classes for the execution process, as well as the classes which are allowed to connected with others, have some common features:

  1. They are all the sub-classes of GPUImageOutput except GPUImageView;
  2. They all obey the protocol of GPUImageInput, except the input classes (GPUImage provides five kinds of input classes which are GPUImagePicture、GPUImageRawDataInput、GPUImageMovie、GPUImageUIElement、GPUImageVideoCamera. It’s easy to know what types of input these classes can deal with).

The class ‘GPUImageOutput‘ is not used directly during the whole process, because all output classes used are its sub-classes. It can be noticed from their names that the general use of GPUImageOutput is to be used as a output and the classes following the GPUImageInput protocol is used as an input. So during the process of connecting each piece of pipeline:

  1. The first object is only as an input, so it must be one of the five input classes and also not follow the GPUImageInput protocol;
  2. The classes which need to be connected in the middle of pipeline, like GPUImageFilter, as well as the father class of all filter classes, not only is the sub-class of GPUImageOutput but also obey the protocol of GPUImageInput. This is because it needs to reveive data from its privous node, and transmit the data to the next node after processing done;
  3. As the last node/object, GPUImageView is not necessary to be the sub-class of GPUImageOutput, in order to no next node at all.

Those are three key points of GPUImage I suppose. Actually I wrote this post in Chinese almost 2 years ago. You see, time is always so fast. And I’ll keep translating other posts about GPUImage in English and hope if there is some chances that I can modify GPUImage or develop a new image processing framework which could implement more interesting effects.

使用CGBitmapContextCreate创建绘制图片的上下文

函数:

CGContextRef CGBitmapContextCreate (

void *data,

size_t width,

size_t height,

size_t bitsPerComponent,

size_t bytesPerRow,

CGColorSpaceRef colorspace,

CGBitmapInfo bitmapInfo

);

参数:

  • data 指向要渲染的绘制内存的地址。这个内存块的大小至少是(bytesPerRow*height)个字节。使用时可填NULL或unsigned char类型的指针。
  • width bitmap的宽度,单位为像素
  • height bitmap的高度,单位为像素
  • bitsPerComponent 内存中像素的每个组件的位数.例如,对于32位像素格式和RGB 颜色空间,你应该将这个值设为8。
  • bytesPerRow bitmap的每一行在内存所占的比特数,一个像素一个byte。
  • colorspace bitmap上下文使用的颜色空间。
  • bitmapInfo 指定bitmap是否包含alpha通道,像素中alpha通道的相对位置,像素组件是整形还是浮点型等信息的字符串。

应用

1.RGBA:

const CGSize size = size;
const size_t bitsPerComponent = 8;
const size_t bytesPerRow = size.width * 4; 
CGBitmapContextCreate(calloc(sizeof(unsigned char), bytesPerRow * size.height), size.width, size.height, bitsPerComponent, bytesPerRow, CGColorSpaceCreateDeviceRGB(), kCGImageAlphaPremultipliedLast);

2.only alpha

const CGSize size              = size;
const size_t bitsPerComponent  = 8;
const size_t bytesPerRow       = size.width; 
CGContextRef context = CGBitmapContextCreate(calloc(sizeof(unsigned char), bytesPerRow * size.height), size.width, size.height, bitsPerComponent, bytesPerRow, NULL, kCGImageAlphaOnly);

局部变量引起的循环引用

开发中写了个demo,具体片段如下,创建了三个局部对象Button,通过RAC做相应的按钮事件操作。没想到竟然发现到有内存泄漏的问题。因为已经排除了RAC导致的原因(具体ReactiveCocoa的内存管理可以看:http://blog.csdn.net/y_csdnblog_xx/article/details/51483111),所以刚开始时候也比较一头雾水。但是最后发现代码中犯了一个很低级的错误。

UIButton * btn0 = [[UIButton alloc]initWithFrame:CGRectMake(100, 450, 50, 100)];
[self.view addSubview:btn0];
UIButton * btn1 = [[UIButton alloc]initWithFrame:CGRectMake(200, 450, 50, 100)];
[self.view addSubview:btn1];
[[btn0 rac_signalForControlEvents:UIControlEventTouchUpInside] subscribeNext:^(UIButton * sender) {

//do something...

btn0.selected = YES;

btn1.selected = NO;

}];
[[btn1 rac_signalForControlEvents:UIControlEventTouchUpInside] subscribeNext:^(UIButton * sender) {

//do something...

btn0.selected = NO;

btn1.selected = YES;

}];

正常我们使用RAC或者Block,如果有全局属性在Block中调用并且该Block被本类对象持有,我们都会对self做一个弱引用,更保险一些的话还会在Block中再强引用一下这个指向self的弱引用指针。这个操作就能很有效避免循环引用带来的内存问题。

但是在以上代码中,我并没有创建全局属性并且也没有在Block中调用任何全局属性,结果同样出现了循环引用。我已经意识到了问题肯定出在Block中,就顺着代码排查可能性。我先把btn1的事件响应信号的接收Block中的btn0.selected = NO;去掉,发现问题解决了。因此意识到,即使是局部变量对象,如果存在同样的情况,也会导致循环引用。

具体说明:经过以上代码,有四个地方引用了btn0对象:1.btn0指针。2.self.view。3.btn0自身事件响应信号接收方法的block。4.btn1自身事件响应信号接收方法的block。同样有四个地方引用了btn1对象:1.btn1指针。2.self.view。3.btn1自身事件响应信号接收方法的block。4.btn0自身事件响应信号接收方法的block。这两按钮前三个情况都没问题,关键在于第四种情况。btn0的释放取决于btn1的释放,btn1同样要等btn0释放后才可释放其block进而才可释放。此时就出现了循环引用。

解决办法:给会导致循环引用的局部对象创建弱引用指针供相应Block中使用。

__weak UIButton * weakBtn0 = btn0;
__weak UIButton * weakBtn1 = btn1;

Category中实现交换本类dealloc方法

项目中不恰当使用某个三方依赖导致了内存问题,想通过查看库中某对象的init和dealloc的具体结果,但是又不能使用@selector()获取dealloc方法,error是ARC禁止这么做。

但发现还有其他办法可以获取到原本的dealloc方法,以下是具体操作:

Method originalDealloc = class_getInstanceMethod(self, NSSelectorFromString(@"dealloc"));

Method swizzledDealloc = class_getInstanceMethod(self, @selector(ac_dealloc));

method_exchangeImplementations(originalDealloc, swizzledDealloc);
- (void)ac_dealloc {

[self ac_dealloc];

NSLog(@"buffer dealloc:%@",self);

}

但个人认为此操作可能会导致上架审核出现问题,因此最好只是用作调试。

更正:交换方法中添加的自定义操作一定要在[self ac_dealloc]之前。

- (void)ac_dealloc {

//do something

NSLog(@"buffer dealloc:%@",self);

[self ac_dealloc];

}

补充:如果想完全重写本类的dealloc方法的话,直接在分类中重写dealloc,这样的话在这个类的对象释放时只会走分类中重写的dealloc方法,而不会走本类中的dealloc。如果想在本类中的dealloc方法实现的内容的基础上添加内容,那就要使用上方“更正”中的代码所示的写法。

子类中调用父类的私有方法

对于在使用cocoapods中,可能三方库中的类并不能满足实际需求,因此我们要对这些类进行子类化或者category等操作。假设我们在子类中想调用父类的方法,对于父类的.m文件内容可见的情况即已经知道具体想调用的方法名时,可以:

第一种最简单的办法

if ([super respondsToSelector:@selector(methodName)]) {

[super performSelector:@selector(methodName)];

}

第二种可以通过runtime获取父类的该方法

Method oriM = class_getInstanceMethod([super class], @selector(methodName));

SEL selector = method_getName(oriM);

[super respondsToSelector:selector];

如会出现“undeclared selector”的warning时

1.可在当前文件文件或.pch中导入

#pragma GCC diagnostic ignored "-Wundeclared-selector"

2.如下在警告的代码前后加上

#pragmaclang diagnostic push
#pragmaclang diagnostic ignored"-Warc-performSelector-leaks"

warning code...

#pragmaclang diagnostic pop

UICollectionView prefetch

iOS10中,UICollectionView和UITableView增加了一个协议属性prefetchDataSource。与dataSource属性相似,将对象赋值给列表的prefetchDataSource属性,并遵循相应协议,即UICollectionViewDataSourcePrefetching,即可在该对象中实现协议中的两方法,从而达到列表内容预加载的效果。值得注意的是,iOS10中列表的prefetchingEnabled默认为YES,如果用自己的办法优化(比如每个cell内容都使用异步加载的方式),那需要将此属性设置为NO:

if ([_inputSelectView respondsToSelector:@selector(setPrefetchingEnabled:)])
{
    _inputSelectView.prefetchingEnabled  = NO;
}

举个例子:

@interface NLViewController <UICollectionViewDelegate, UICollectionViewDataSource, UICollectionViewDelegateFlowLayout, UICollectionViewDataSourcePrefetching>

@end

static NSString *cellIdentifier = @"cell";

@implementation NLViewController

#pragma mark - View lifeCycle

- (void)viewDidLoad {
[super viewDidLoad];

collectionView.delegate                      = self;
collectionView.dataSource                    = self;
collectionView.prefetchDataSource            = self;
}

当滑动列表速度超过cellForItemAtIndexPaths承受数据加载能力时候,会调用该方法。因此,列表初始化时、滑的很慢时都不会调用,其中indexPaths参数为即将准备进入可视区域的cell对应的indexPath数组。

- (void)tableView:(UITableView *)tableViewprefetchRowsAtIndexPaths:(NSArray*)indexPaths {    

for (NSIndexPath *indexPath in indexPaths) {

//请求或处理图片

    }

}

从一定角度来看,collection view 的预加载请求只是试图优化未来不确定状态的一种猜测,这种状态可能并不会真实发生。例如,如果滚动速度放缓或者完全反转方向,那些已经请求过的预加载 cell 可能永远都不会确切地显示。

- (void)tableView:(UITableView *)tableViewcancelPrefetchingForRowsAtIndexPaths:(NSArray *)indexPaths;

NSNotification object

我们正常使用NSNotification进行消息传递的时候,附带参数都是通过userInfo来实现。但是最近试了一下,例如我想传个BOOL类型的参数,试着用object来传递也是可以实现的。试了以下几种情况:

  • post填nil, add填XXX。此时收不到通知。
  • post填XXX, add填nil。此时收到通知。
  • post填XXX, add填XXX。此时收到通知。
  • post填nil, add填nil。此时收到通知。
[[NSNotificationCenter defaultCenter] addObserver:self selector:@selector(XXXX:) name:@"XXXXX" object:objectA];
[[NSNotificationCenter defaultCenter] postNotificationName:@"XXXXX" object:objectA userInfo:@{@"key":@"param"}];

可是按道理object这个参数并不是用来做参数传递的。翻了一下官方文档:官方解释最后一个参数为notificationSender,解释如下:

The object whose notifications the observer wants to receive; that is, only notifications sent by this sender are delivered to the observer. When nil, the notification center doesn’t use a notification’s sender to decide whether to deliver it to the observer.

官方并没有对object参数做限制,但是正确使用NSNotification的姿势应该是所有想进行传递的参数都应该放在userInfo中,object只作为收发通知的一个限制要求。

UIView与CALayer

因为现在手头上的项目而着重研究了一下CALayer的使用,发现这个类完全就是UI实现的新大陆。在之前项目中以及面试开发时发现,正常做常规社交或者信息展示类app的开发同学接触到的CALayer也只局限于设置个圆角边框之类(可能相比之下我之前的知识面比较狭小?)。总之,深入学习了CALayer之后会更深一步领悟到UI控件各种属性的真谛,比如新手困惑frame跟bounds跟transform的关系。更能实现一些意想不到的UI效果,比如展示控件的3D效果、遮盖效果以及渐变、粒子啦,现在开发的项目的实现核心功能正是通过CALayer的实现来完成。(部分内容为转载)

  1. CALayer包含在QuartzCore框架中,这是一个跨平台的框架,既可以用在iOS中又可以用在Mac OS X中。

2. 在iOS中CALayer的设计主要是了为了内容展示和动画操作,CALayer本身并不包含在UIKit中,它不能响应事件。所以在使用过程中,需要通过手势来控制CAlayer的属性变化(transform啦等等)时需要用一个view来接收手势(touch事件或者GuestureRecognizer),并在事件响应方法中实现CAlayer的变化。

3. 由于CALayer在设计之初就考虑它的动画操作功能,CALayer很多属性在修改时都能形成动画效果,这种属性称为“隐式动画属性”。但是对于UIView的根图层(view.layer)而言属性的修改并不形成动画效果,因为很多情况下根图层更多的充当容器的做用,如果它的属性变动形成动画效果会直接影响子图层。

4. UIView负责创建并管理CALayer。比如一个:viewA上add了一个viewB,则viewB.layer也成为了viewA.layer的sublayer。

5. CALayer的各种属性:

  • contents id类型,就是啥类型都能对他赋值都不会报错,但是赋值完看上去有效果的只有图片类型。IOS开发中具体实现为:view.layer.contents = (id)[UIImage imageNamed:”XXX”].CGImage; 听说MAC开发中需要用NSImage。UIImageView内部就使用这个东西实现图片显示。
  • contentGravity = view.contentMode
  • contentsScale = view.contentScaleFactor
  • maskToBounds = view.clipToBounds
  • contentsRect 跟contents有关系的属性,就是规定了contents的可视区域。但是通过这个设置的可视区域只能为一个矩形区域。
  • contentsCenter CGRect类型,规定了contents的放大区域,当设置contentsScale时这个区域就变大啦,以外区域就缩小啦

6. (这段完全是载的)CGImage并不是唯一可以赋值给contents属性的,也可以使用Core Graphics绘制寄宿图给它,如果你实现了drawRect方法,然后如果你调用setNeedsDisplay或者外观属性被改变时,它就会自动调用drawRect自动重绘,虽然drawRet是一个UIView方法,但是其实都是底层都是CALayer重绘保存了图片,如果你不需要自定义绘制就不要写一个空的drawRect方法,它很消耗cpu和内存资源,CALayer有一个可选的delegate属性,如果设置了delegate,并主动调用了layer的displey方法(注意和drawRect不同这个重绘时机是开发者自己控制的,也可以调用setNeedsDisplay方法给系统自己找时机调用),它会调用displayLayer方法,在这里是设置contents属性的最后机会了,如果你没有实现这个方法,它会尝试去调用下面这个方法:drawLayer。。。,如果你实现了displayLayer方法,drawLayer就不会调用了

7. 其实!UIView的frame、bounds、center、transform这些属性只是存取方法!真正实现UIView显示出来的样子的都是通过这个控件的layer来实现。所以

view.center = view.layer.position 

为对应父图层的anchorPoint的所在位置,也就是该view在他的父视图坐标中的view的中心点位置。(这段也是载的)CALayer通过anchorPoint(锚点)和center(position)对齐来控制UIView的位置,锚点是相对UIView的一个位置,而center就是一个点,由于anchorPoint属性对UIView是屏蔽的,而anchorPoint默认值又是{0.5,0.5},所以这个属性才叫center.而UIView和CALayertransform旋转也是围绕这anchorPoint旋转的。

view.frame = view.layer.frame    

frame是通过bounds、position、transform来得出的,比如说view通过transform放大了2倍,那frame中的width跟height就比bounds中的大了2倍,所以进行transform后frame不一定等于bounds

view.bounds = view.layer.bounds
view.transform = view.layer.CGAffineTransform 

layer共有两个transform的属性:CGAffineTransform、CATransform3D,我的理解是CGAffineTransform是CATransform3D的一个平面映射,这里有很深的套路啊,有时间专门再写一篇transform的理解跟我在使用时遇到的问题。

其他layer有关位置的属性:

  • geometryFlipped 这个用来翻转layer所在的坐标系,上下翻!就是!y轴负的跟正的翻!
  • zPosition 看名字就能明白是用来设置layer的层级位置的,默认为0,越大越往上!

8. 关于layer的自适应:

UIView在做自适应时候我们可以采取autoresizingMask、autoLayout或者巨好用的masonry来实现。但是!IOS平台的Layer并不能像View这样优雅地自适应!听说只有MAC开发才行。。。所以导致我在开发项目过程中不能全程使用masonry。。。想做ios中layer层的自适应也是有办法的,比如你可以遵循layer的代理,然后在layoutSublayersOfLayer代理方法中实现,这样当然免不了一堆的判断跟计算啦。