铺垫和引入

encoder使用VGG-16的卷积层进行学习，decoder使用反卷积deconv和反池化unpool进行上采样。

将object proposal候选域推荐（edge box画方框）送入训练后的网络，整幅图像是这些proposal分割结果的组合，这样就可以解决物体太大或者太小所带来的分割问题，改进了现存基于FCN的方法

? ? ? ? 但是这样做需要人为干预，网络从全自动变成半自动的，我们其实不推荐这样改。

FCN的critical limitations

感受野大小固定predefined fixed-size?receptive field。——The object that is substantially larger or smaller than the receptive field may be fragmented支离破碎的-割开?or mislabeled
1. （1）有的太大的物体，我的感受野如果没那么大的话，会导致分割错误，因为我们没有把整个大物体都看全，我就会认为整个大物体是由两个小物体组成的（意思就是你把一个东西给割成了两个物体），从而造成错分。大的object只会用到局部信息Label prediction is done with only local information for large objects and相同标签的object可能会被赋上不同的标签?the pixels that belong to the same object may have inconsistent不同的-不一致的 label。你可以看原文提供的下图，你的感受野比较小，object很大，图中好几个小物体识别不出来，train bycycle persion各自的像素点都被label成了bus
2. （2）尺度太小的物体，由于我的感受野是比较大的，看到的物体的背景信息和其他信息比较多，进而影响我对这个小物体的判断，我很容易把这些小物体当做background。Small objects are often ignored and classified as background.图中这些人太小了，FCN误把他们当做background上的几个纹理了，没识别出来。
?这个特征图经过很多次下采样以后就变得很模糊了、像素很低了、尺寸太小了，在这个过程中物体的精细结构和细节经常被丢失，然后你再把这个很模糊的特征图送给反卷积层进上采样恢复，很难做到精细分割这个效果

模型结构DeconvNet

? Encoder和Decoder部分是在模仿VGG-16

? ? ? ? Encoder(它起名叫Convolution network)：卷积-卷积-池化”两次，“卷积-卷积-卷积-池化”三次

? ? ? ? 中间是两个全连接层，其实我个人觉得没必要，一是大大增大的参数量，二是一个二维的图变成一维向量没啥意义，对performance的提高其实很有限

? ? ? ? Decoder(它起名叫Deconvolution network):“上采样-卷积-卷积-卷积”三次，“上采样-卷积-卷积”两次

? ? ? ? 一个softmax分类层

池化的作用：

? ? ? ? 过滤上一层的噪音，如何做到过滤呢？用感受野内一个单独的代表值作为激活来提取想要的信息Pooling in convolution network is designed to filter noisy actications in a lower layer by abstracting提取、摘要、抽象化?activations in a receptive field with a single representative value

? ? ? ? 但是池化也丢失了一部分的空间信息。spatial information within a receptive filed is lost during pooling

代码实现

各层网络的参数和尺寸表

import torch
import torchvision.models as models
from torch import nn


vgg16_pretrained = models.vgg16(pretrained=False)


def decoder(input_channel, output_channel, num=3):
    if num == 3:
        decoder_body = nn.Sequential(
            nn.ConvTranspose2d(input_channel, input_channel, 3, padding=1),
            nn.ConvTranspose2d(input_channel, input_channel, 3, padding=1),
            nn.ConvTranspose2d(input_channel, output_channel, 3, padding=1))
    elif num == 2:
        decoder_body = nn.Sequential(
            nn.ConvTranspose2d(input_channel, input_channel, 3, padding=1),
            nn.ConvTranspose2d(input_channel, output_channel, 3, padding=1))

    return decoder_body


class VGG16_deconv(torch.nn.Module):
    def __init__(self):
        super(VGG16_deconv, self).__init__()

        pool_list = [4, 9, 16, 23, 30]
        for index in pool_list:
            vgg16_pretrained.features[index].return_indices = True

        self.encoder1 = vgg16_pretrained.features[:4]
        self.pool1 = vgg16_pretrained.features[4]

        self.encoder2 = vgg16_pretrained.features[5:9]
        self.pool2 = vgg16_pretrained.features[9]

        self.encoder3 = vgg16_pretrained.features[10:16]
        self.pool3 = vgg16_pretrained.features[16]

        self.encoder4 = vgg16_pretrained.features[17:23]
        self.pool4 = vgg16_pretrained.features[23]

        self.encoder5 = vgg16_pretrained.features[24:30]
        self.pool5 = vgg16_pretrained.features[30]

        self.classifier = nn.Sequential(
            torch.nn.Linear(512 * 11 * 15, 4096),
            torch.nn.ReLU(),
            torch.nn.Linear(4096, 512 * 11 * 15),
            torch.nn.ReLU(),
        )

        self.decoder5 = decoder(512, 512)
        self.unpool5 = nn.MaxUnpool2d(2, 2)

        self.decoder4 = decoder(512, 256)
        self.unpool4 = nn.MaxUnpool2d(2, 2)

        self.decoder3 = decoder(256, 128)
        self.unpool3 = nn.MaxUnpool2d(2, 2)

        self.decoder2 = decoder(128, 64, 2)
        self.unpool2 = nn.MaxUnpool2d(2, 2)

        self.decoder1 = decoder(64, 12, 2)
        self.unpool1 = nn.MaxUnpool2d(2, 2)

    def forward(self, x):                       # 3, 352, 480
        encoder1 = self.encoder1(x)             # 64, 352, 480
        output_size1 = encoder1.size()          # 64, 352, 480
        pool1, indices1 = self.pool1(encoder1)  # 64, 176, 240

        encoder2 = self.encoder2(pool1)         # 128, 176, 240
        output_size2 = encoder2.size()          # 128, 176, 240
        pool2, indices2 = self.pool2(encoder2)  # 128, 88, 120

        encoder3 = self.encoder3(pool2)         # 256, 88, 120
        output_size3 = encoder3.size()          # 256, 88, 120
        pool3, indices3 = self.pool3(encoder3)  # 256, 44, 60

        encoder4 = self.encoder4(pool3)         # 512, 44, 60
        output_size4 = encoder4.size()          # 512, 44, 60
        pool4, indices4 = self.pool4(encoder4)  # 512, 22, 30

        encoder5 = self.encoder5(pool4)         # 512, 22, 30
        output_size5 = encoder5.size()          # 512, 22, 30
        pool5, indices5 = self.pool5(encoder5)  # 512, 11, 15

        pool5 = pool5.view(pool5.size(0), -1)
        fc = self.classifier(pool5)
        fc = fc.reshape(1, 512, 11, 15)

        unpool5 = self.unpool5(input=fc, indices=indices5, output_size=output_size5)    # 512, 22, 30
        decoder5 = self.decoder5(unpool5)   # 512, 22, 30

        unpool4 = self.unpool4(input=decoder5, indices=indices4, output_size=output_size4)  # 512, 44, 60
        decoder4 = self.decoder4(unpool4)   # 256, 44, 60

        unpool3 = self.unpool3(input=decoder4, indices=indices3, output_size=output_size3)  # 256, 88, 120
        decoder3 = self.decoder3(unpool3)   # 128, 88, 120

        unpool2 = self.unpool2(input=decoder3, indices=indices2, output_size=output_size2)  # 128, 176, 240
        decoder2 = self.decoder2(unpool2)  # 64, 176, 240

        unpool1 = self.unpool1(input=decoder2, indices=indices1, output_size=output_size1)  # 64, 352, 480
        decoder1 = self.decoder1(unpool1)  # 12, 352, 480

        return decoder1


if __name__ == "__main__":
    import torch as t

    rgb = t.randn(1, 3, 352, 480)

    net = VGG16_deconv()

    out = net(rgb)

    print(out.shape)