前言
前几天了解了3D深度学习的入门之作：PointNet(具体原理解释可以点击链接看另一篇博客)，今天为Ponint++的理解做篇笔记。最好结合原理和代码来看可以理解的更加深刻。
论文：PointNet++
代码：Pytorch-PointNet++
提出问题
前篇PointNet是对每一个点做从低维到高维的映射的学习，缺点就是没有局部的概念，要么对一个点的操作要么是对所有点的操作，很难对更加精细复杂的场景进行分析。所以作者在PointNet++上做了改进使网络能够更好的提取局部特征分析。
- 利用空间距离，使用PointNet对点集局部区域进行特征迭代提取，使其学到局部尺度越来越大的特征。
- 针对点集分布不均匀的问题，提出了新的集合学习层，以自适应的组合来自多个尺度的特征。
点云分布不一致时，每个子区域中如果在生成的时候使用相同的半径r，会导致有些区域采样点过少。所以文章提出了两种方法：Multi-scale grouping (MSG) and Multi-resolution grouping (MRG)
MSG是一种多尺度分组，就是一个中心点，根据不同的尺度生成不同范围的局部区域，每个区域的半径及里面的点的个数不同。中间的实线和虚线表示用PointNet进行特征的提取，之后在concat作为这个中心点的特征。也就是说MSG实际上相当于并联了多个hierarchical structure，每个结构中心点不变，但是区域范围不同。
MRG是一种多分辨率分组。就是对不同层的分组进行concat，低层的区域进行特征提取再和高层的区域进行concat。— ResNet的跳跃连接类似
网络结构
从图中可以看出，作者在论文中提出了一种SA（set abstraction）层，多层次结合逐步提取更多的局部特征。每个SA层主要由三部分组成：sampling layer, grouping layer, PointNet layer。
Sampling layer: 对输入点进行采样，选取若干中心点。
Grouping layer: 利用中心点划分局部区域。
PointNet layer: 作为特征提取器提取生成的局部特征，类似于CNN中卷积块作为特征提取器，
对应的区域都是n*n的像素区域。在3D点云中，局部区域是生成的球型区域，用PointNet提取区域。
1. Sampling layer
采样层顾名思义就是从输入的点云数据中随机采样，采用的方法是FPS(arthest point sampling),相比于随机采样，这种方法能更好的覆盖整个点集。
FPS算法原理：
1、 从点云中选取第一个点A作为查询点，从剩余点中，选取一个距离最远的点B；
2、以取出来的点A，B作为查询点，从剩余点中，取距离最远的点C。此时，由于已经取出来的点的个数超过1，需要同时考虑所有查询点（A，B）。
方法如下：
对于剩余点中的任意一个点P，计算该点P到已经选中的点集中所有点（A, B）的距 离；取与点A和B的距离最小值作为该点到已选点集的距离d；再计算出每个剩余点到点集的距离后，选取距离最大的那个点，即为点C。
3、 重复第2步，一直采样到N’个点为止。
该方法的pytorch代码如下：
def farthest_point_sample(xyz, npoint):
"""
Input:
xyz: pointcloud data, [B, N, 3]
npoint: number of samples
Return:
centroids: sampled pointcloud index, [B, npoint]
"""
device = xyz.device
B, N, C = xyz.shape # [6, 2048, 3]
centroids = torch.zeros(B, npoint, dtype=torch.long).to(device) # [6, 1024]。定义中心矩阵，npoint是采样数
distance = torch.ones(B, N).to(device) * 1e10 # [6, 2048]。 全局矩阵，N是点云总数2048
farthest = torch.randint(0, N, (B,), dtype=torch.long).to(device) # [6]在0-N范围内随机定义最远点矩阵，尺寸是【batch_size】
batch_indices = torch.arange(B, dtype=torch.long).to(device) # [6] = [0, 1, 2, 3, 4, 5]。定义batch size的索引
for i in range(npoint):
centroids[:, i] = farthest。 #选取最远点赋给中心矩阵作为采样点，对应上述步骤1
centroid = xyz[batch_indices, farthest, :].view(B, 1, 3) # [6, 1 ,3] 提取这个最远点的xyz空间坐标信息
dist = torch.sum((xyz - centroid) ** 2, -1) # [6, 2048] 计算选取的最远点到所有点的欧式距离
mask = dist < distance # [6, 2048]。更新distances，记录样本中每个点距离所有已出现的采样点的最小距离。
distance[mask] = dist[mask]
farthest = torch.max(distance, -1)[1] # 寻找最远点作为下一次迭代使用
return centroids
2、Grouping layer
利用上一步生成的中心点矩阵，结合ball query 方法生成空间局部区域。重要的参数有两个，一个是区域中点的个数，另一个是球的半径。
def query_ball_point(radius, nsample, xyz, new_xyz):
"""
Input:
radius: local region radius
nsample: max sample number in local region
xyz: all points, [B, N, 3]
new_xyz: query points, [B, S, 3]
Return:
group_idx: grouped points index, [B, S, nsample]
"""
device = xyz.device
B, N, C = xyz.shape
_, S, _ = new_xyz.shape
group_idx = torch.arange(N, dtype=torch.long).to(device).view(1, 1, N).repeat([B, S, 1]) # [B,S,N]=[6, 1024, 2048]
sqrdists = square_distance(new_xyz, xyz) # [6, 1024, 2048]. 中心点与所有点之间的欧式距离
group_idx[sqrdists > radius ** 2] = N # 将距离大于 r^2 的点赋值为N， 其余值保留。
group_idx = group_idx.sort(dim=-1)[0][:, :, :nsample] # [6, 1024, 32]生序排列，取出前面的nsample个样本数
#考虑到有可能前nsample个点中也有被赋值为N的点（即球形区域内不足nsample个点），这种点需要舍弃，直接用第一个点来代替即可。
#group_first: [B, S, nsample]， 实际就是把group_idx中的第一个点的值复制为了[B, S, nsample]的维度，便利于后面的替换
group_first = group_idx[:, :, 0].view(B, S, 1).repeat([1, 1, nsample]) # [6, 1024, 32]
mask = group_idx == N # [6, 1024, 32] 找到group_idx中值等于N的点，将这些点的值替换为第一个点的值
group_idx[mask] = group_first[mask]
return group_idx
之后将所有的点云采样分组，便于下一步的特征提取。有sample_and_group和sample_and_group_all两个函数，其区别在于sample_and_group_all直接将所有点作为一个group。
def sample_and_group(npoint, radius, nsample, xyz, points, returnfps=False):
"""
Input:
npoint:
radius:
nsample:
xyz: input points position data, [B, N, 3]
points: input points data, [B, N, D]
Return:
new_xyz: sampled points position data, [B, npoint, nsample, 3]
new_points: sampled points data, [B, npoint, nsample, 3+D]
"""
B, N, C = xyz.shape # 6, 2048, 3
S = npoint # 1024
fps_idx = farthest_point_sample(xyz, npoint) # [6, 1024]采样点索引
new_xyz = index_points(xyz, fps_idx) # [6, 1024, 3] 将采样点从所有的点中提取出来形成new_xyz。
idx = query_ball_point(radius, nsample, xyz, new_xyz) # [6, 1024, 32]生成局部区域的索引，有S个中心，每个有nsample个采样点的索引。
grouped_xyz = index_points(xyz, idx) # [B, npoint, nsample, C] [6, 1024, 32, 3]根据索引从所有数据中挑选出分组区域
grouped_xyz_norm = grouped_xyz - new_xyz.view(B, S, 1, C)# 每个区域的点减去区域的中心值
# 如果每个点上面有新的特征的维度，则用新的特征与旧的特征拼接，否则直接返回旧的特征
if points is not None:
grouped_points = index_points(points, idx) # [6, 1024, 32, 9]
new_points = torch.cat([grouped_xyz_norm, grouped_points], dim=-1) # [B, npoint, nsample, C+D]=[6, 1024, 32, 12]
else:
new_points = grouped_xyz_norm
if returnfps:
return new_xyz, new_points, grouped_xyz, fps_idx
else:
return new_xyz, new_points
def sample_and_group_all(xyz, points):
"""
Input:
xyz: input points position data, [B, N, 3]
points: input points data, [B, N, D]
Return:
new_xyz: sampled points position data, [B, 1, 3]
new_points: sampled points data, [B, 1, N, 3+D]
"""
device = xyz.device
B, N, C = xyz.shape
new_xyz = torch.zeros(B, 1, C).to(device)
grouped_xyz = xyz.view(B, 1, N, C)
if points is not None:
new_points = torch.cat([grouped_xyz, points.view(B, 1, N, -1)], dim=-1)
else:
new_points = grouped_xyz
return new_xyz, new_points
3、PointNet layer
PointNet layer就是将选取的局部区域进行特征的提取。首先先通过sample_and_group的操作形成局部的group，然后对局部的group中的每一个点做MLP操作，最后进行局部的最大池化，得到局部的全局特征。
class PointNetSetAbstraction(nn.Module):
def __init__(self, npoint, radius, nsample, in_channel, mlp, group_all): # [1024,0.1,32,12,[32,32,64],false]
super(PointNetSetAbstraction, self).__init__()
self.npoint = npoint
self.radius = radius
self.nsample = nsample
self.mlp_convs = nn.ModuleList()
self.mlp_bns = nn.ModuleList()
last_channel = in_channel
for out_channel in mlp:
self.mlp_convs.append(nn.Conv2d(last_channel, out_channel, 1))
self.mlp_bns.append(nn.BatchNorm2d(out_channel))
last_channel = out_channel
self.group_all = group_all
def forward(self, xyz, points):
"""
Input:
xyz: input points position data, [B, C, N]
points: input points data, [B, D, N]
Return:
new_xyz: sampled points position data, [B, C, S]
new_points_concat: sample points feature data, [B, D', S]
"""
xyz = xyz.permute(0, 2, 1) # [6, 3, 2048] --- [6, 2048, 3]
if points is not None:
points = points.permute(0, 2, 1) # [6, 2048, 9]
if self.group_all:
new_xyz, new_points = sample_and_group_all(xyz, points)
else:
new_xyz, new_points = sample_and_group(self.npoint, self.radius, self.nsample, xyz, points)
# new_xyz: sampled points position data, [B, npoint, C] [6, 1024, 3]
# new_points: sampled points data, [B, npoint, nsample, C+D] [6, 1024, 3+9]
new_points = new_points.permute(0, 3, 2, 1) # [B, C+D, nsample,npoint] [6, 12, 32, 1024]
#利用1x1的2d的卷积相当于把每个group当成一个通道，共npoint个通道，对[C+D, nsample]的维度上做逐像素的卷积，结果相当于对单个C+D维度做1d的卷积。
for i, conv in enumerate(self.mlp_convs):
bn = self.mlp_bns[i]
new_points = F.relu(bn(conv(new_points)))
# 对每个group做一个max pooling得到局部的全局特征
new_points = torch.max(new_points, 2)[0] # [6, 64, 1024]
new_xyz = new_xyz.permute(0, 2, 1) # [6, 1024, 3] -- [ 6, 3, 1024]
return new_xyz, new_points
SA层加入MSG后的代码类似，只不过输入的半径不仅仅是一个了，而是一个列表，然后做ball_quary,最终将不同半径下的点点云特征保存在new_points_list中，再最后拼接到一起。
class PointNetSetAbstractionMsg(nn.Module):
def __init__(self, npoint, radius_list, nsample_list, in_channel, mlp_list):
super(PointNetSetAbstractionMsg, self).__init__()
self.npoint = npoint
self.radius_list = radius_list
self.nsample_list = nsample_list
self.conv_blocks = nn.ModuleList()
self.bn_blocks = nn.ModuleList()
for i in range(len(mlp_list)):
convs = nn.ModuleList()
bns = nn.ModuleList()
last_channel = in_channel + 3
for out_channel in mlp_list[i]:
convs.append(nn.Conv2d(last_channel, out_channel, 1))
bns.append(nn.BatchNorm2d(out_channel))
last_channel = out_channel
self.conv_blocks.append(convs)
self.bn_blocks.append(bns)
def forward(self, xyz, points):
"""
Input:
xyz: input points position data, [B, C, N]
points: input points data, [B, D, N]
Return:
new_xyz: sampled points position data, [B, C, S]
new_points_concat: sample points feature data, [B, D', S]
"""
xyz = xyz.permute(0, 2, 1)
if points is not None:
points = points.permute(0, 2, 1)
B, N, C = xyz.shape
S = self.npoint
new_xyz = index_points(xyz, farthest_point_sample(xyz, S))
new_points_list = []
for i, radius in enumerate(self.radius_list):
K = self.nsample_list[i]
group_idx = query_ball_point(radius, K, xyz, new_xyz)
grouped_xyz = index_points(xyz, group_idx)
grouped_xyz -= new_xyz.view(B, S, 1, C)
if points is not None:
grouped_points = index_points(points, group_idx)
grouped_points = torch.cat([grouped_points, grouped_xyz], dim=-1)
else:
grouped_points = grouped_xyz
grouped_points = grouped_points.permute(0, 3, 2, 1) # [B, D, K, S]
for j in range(len(self.conv_blocks[i])):
conv = self.conv_blocks[i][j]
bn = self.bn_blocks[i][j]
grouped_points = F.relu(bn(conv(grouped_points)))
new_points = torch.max(grouped_points, 2)[0] # [B, D', S]
new_points_list.append(new_points)
new_xyz = new_xyz.permute(0, 2, 1)
new_points_concat = torch.cat(new_points_list, dim=1)
return new_xyz, new_points_concat
细节部分已经讲完，接下来就是整体的主体部分。
如果是分类任务，整体结构看上图的下部分。实际进行了两个SA层进行局部的特征提取，得到更加精细的特征，整体上和CNN类似。第三个SA其实就是对所有的点的特征提取，所以group_al l= True。再进行全连接层最终得到分类。
class get_model(nn.Module):
def __init__(self,num_class,normal_channel=True):
super(get_model, self).__init__()
in_channel = 3 if normal_channel else 0
self.normal_channel = normal_channel
self.sa1 = PointNetSetAbstractionMsg(512, [0.1, 0.2, 0.4], [16, 32, 128], in_channel,[[32, 32, 64], [64, 64, 128], [64, 96, 128]])
self.sa2 = PointNetSetAbstractionMsg(128, [0.2, 0.4, 0.8], [32, 64, 128], 320,[[64, 64, 128], [128, 128, 256], [128, 128, 256]])
self.sa3 = PointNetSetAbstraction(None, None, None, 640 + 3, [256, 512, 1024], True)
self.fc1 = nn.Linear(1024, 512)
self.bn1 = nn.BatchNorm1d(512)
self.drop1 = nn.Dropout(0.4)
self.fc2 = nn.Linear(512, 256)
self.bn2 = nn.BatchNorm1d(256)
self.drop2 = nn.Dropout(0.5)
self.fc3 = nn.Linear(256, num_class)
def forward(self, xyz):
B, _, _ = xyz.shape
if self.normal_channel:
norm = xyz[:, 3:, :]
xyz = xyz[:, :3, :]
else:
norm = None
l1_xyz, l1_points = self.sa1(xyz, norm)
l2_xyz, l2_points = self.sa2(l1_xyz, l1_points)
l3_xyz, l3_points = self.sa3(l2_xyz, l2_points)
x = l3_points.view(B, 1024)
x = self.drop1(F.relu(self.bn1(self.fc1(x))))
x = self.drop2(F.relu(self.bn2(self.fc2(x))))
x = self.fc3(x)
x = F.log_softmax(x, -1)
return x,l3_points
但是对于分割的任务，我们还需要将点集上采样回原始点集数量，这里使用了分层的差值方法。代码为：
class PointNetFeaturePropagation(nn.Module):
def __init__(self, in_channel, mlp):
super(PointNetFeaturePropagation, self).__init__()
self.mlp_convs = nn.ModuleList()
self.mlp_bns = nn.ModuleList()
last_channel = in_channel
for out_channel in mlp:
self.mlp_convs.append(nn.Conv1d(last_channel, out_channel, 1))
self.mlp_bns.append(nn.BatchNorm1d(out_channel))
last_channel = out_channel
def forward(self, xyz1, xyz2, points1, points2):
"""
Input:
xyz1: input points position data, [B, C, N]
xyz2: sampled input points position data, [B, C, S]
points1: input points data, [B, D, N]
points2: input points data, [B, D, S]
Return:
new_points: upsampled points data, [B, D', N]
"""
xyz1 = xyz1.permute(0, 2, 1) # [6, 3, 64]--- [6, 64, 3 ]
xyz2 = xyz2.permute(0, 2, 1) # [6, 3, 16]---- [6, 16, 3]
points2 = points2.permute(0, 2, 1) # [6, 512, 16]--- [6, 16, 512]
B, N, C = xyz1.shape # [6, 64, 3]
_, S, _ = xyz2.shape # 16
if S == 1:
interpolated_points = points2.repeat(1, N, 1)
else:
dists = square_distance(xyz1, xyz2) # [6, 64, 16]
dists, idx = dists.sort(dim=-1) # [6, 64, 16]
dists, idx = dists[:, :, :3], idx[:, :, :3] # [B, N, 3] [6, 64, 3]
dist_recip = 1.0 / (dists + 1e-8)
norm = torch.sum(dist_recip, dim=2, keepdim=True) # [6, 64, 1]
weight = dist_recip / norm # [6, 64, 3]
interpolated_points = torch.sum(index_points(points2, idx) * weight.view(B, N, 3, 1), dim=2) # [6, 64, 512]
if points1 is not None:
points1 = points1.permute(0, 2, 1) # [6, 64, 256]
new_points = torch.cat([points1, interpolated_points], dim=-1) # [6, 64, 768]
else:
new_points = interpolated_points
new_points = new_points.permute(0, 2, 1) # [6, 768, 64]
for i, conv in enumerate(self.mlp_convs):
bn = self.mlp_bns[i]
new_points = F.relu(bn(conv(new_points)))
return new_points
分割整体的结构为：
class get_model(nn.Module):
def __init__(self, num_classes): # num_classes = 13
super(get_model, self).__init__()
self.sa1 = PointNetSetAbstraction(1024, 0.1, 32, 9 + 3, [32, 32, 64], False)
self.sa2 = PointNetSetAbstraction(256, 0.2, 32, 64 + 3, [64, 64, 128], False)
self.sa3 = PointNetSetAbstraction(64, 0.4, 32, 128 + 3, [128, 128, 256], False)
self.sa4 = PointNetSetAbstraction(16, 0.8, 32, 256 + 3, [256, 256, 512], False)
self.fp4 = PointNetFeaturePropagation(768, [256, 256])
self.fp3 = PointNetFeaturePropagation(384, [256, 256])
self.fp2 = PointNetFeaturePropagation(320, [256, 128])
self.fp1 = PointNetFeaturePropagation(128, [128, 128, 128])
self.conv1 = nn.Conv1d(128, 128, 1)
self.bn1 = nn.BatchNorm1d(128)
self.drop1 = nn.Dropout(0.5)
self.conv2 = nn.Conv1d(128, num_classes, 1)
def forward(self, xyz):
l0_points = xyz # [6, 9, 2048]
l0_xyz = xyz[:,:3,:] # [6, 3, 2048]
l1_xyz, l1_points = self.sa1(l0_xyz, l0_points) # [6, 3, 1024], [6, 64, 1024]
l2_xyz, l2_points = self.sa2(l1_xyz, l1_points) # [6, 3, 256], [6, 128, 256]
l3_xyz, l3_points = self.sa3(l2_xyz, l2_points) # [6, 3, 64], [6, 256, 64]
l4_xyz, l4_points = self.sa4(l3_xyz, l3_points) # [6, 3, 16], [6, 512, 16]
l3_points = self.fp4(l3_xyz, l4_xyz, l3_points, l4_points) # [6, 256, 64]
l2_points = self.fp3(l2_xyz, l3_xyz, l2_points, l3_points) # [6, 256, 256]
l1_points = self.fp2(l1_xyz, l2_xyz, l1_points, l2_points) # [6, 128, 1024]
l0_points = self.fp1(l0_xyz, l1_xyz, None, l1_points) # [6, 128, 2048]
x = self.drop1(F.relu(self.bn1(self.conv1(l0_points))))
x = self.conv2(x) # [6, 128, 2048] ---- [6, 13, 2048]
x = F.log_softmax(x, dim=1)
x = x.permute(0, 2, 1) # [6, 2048, 13]
print(f'x的尺寸：{x.shape}, l4_points的尺寸：{l4_points.shape}')
return x, l4_points
好啦，网络结构已经分析完了，结束。
参考
PointNet++ 论文及代码解读
PointNet++详解与代码
PointNet++具体实现详解----这里面有个运算图也可以帮助理解。
PointNet++的pytorch实现代码阅读