格式简介
libsvm和libfm格式相同。在推荐系统中由于离散特征过于稀疏，为了减小存储负担，只保存索引值，即值为0的不存储。样例格式如下：
<label> <index1>:<value1> <index2>:<value2> ...
...
...
通过全数字的方式节省存储空间。其中，index表示第几个特征（事先记录号对应关系），value表示对应值。如果不在乎特征名称所占用的存储，可以直接存储名称，如下：
<label> <feature1>:<value1> <feature2>:<value2> ...
...
...
该种方式直接存储特征名称和特征值。还可以采用excel形式的csv格式
<label> <feature1> <feature2> ...
<label> <value1> <value2> ...
<label> <value1> <value2> ...
...
...
csv转libsvm格式
读取每条数据，把有值的特征和值组成键值对，重新写入。
data = pd.read_csv('./data/train_1m.txt', header=None, delimiter='\t',)
data.columns = ['label', 'I1', 'I2', 'I3', 'I4', 'I5', 'I6', 'I7', 'I8', 'I9', 'I10', 'I11', 'I12', 'I13',
'C1', 'C2', 'C3', 'C4', 'C5', 'C6', 'C7', 'C8', 'C9', 'C10', 'C11', 'C12', 'C13', 'C14', 'C15',
'C16', 'C17', 'C18', 'C19', 'C20', 'C21', 'C22', 'C23', 'C24', 'C25', 'C26',
]
data = data.fillna('-')
for index, row in data.iterrows():
txt = ''
label = str(row[0])
txt += label
txt += '\t'
for i, fea in enumerate(row[1:]):
if fea != '-':
txt += data.columns[i + 1]
txt += ':'
txt += str(fea)
if i != len(row[1:]) - 1:
txt += '\t'
print(txt)
with open('./data/train_libfm','a') as f:
f.write(txt)
libsvm转tf.record格式
tf.record格式是tensorflow的推荐格式。
def float_feature(value):
return tf.train.Feature(float_list=tf.train.FloatList(value=[value]))
def int64_feature(value):
return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))
def byte_feature(value):
return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value.encode()]))
def line_processor(line):
features = dict()
line = line.strip().split('\t')
label = line[0]
features['label'] = float_feature(float(label))
for i in range(1, 14):
features['I' + str(i)] = float_feature(0)
for i in range(1, 27):
features['C' + str(i)] = byte_feature('0')
sparse_features = ['C' + str(i) for i in range(1, 27)] # 离散特征
dense_features = ['I' + str(i) for i in range(1, 14)] # 连续特征
for fea in line[1:]:
fea_name, fea_value = fea.split(':')
if fea_name in sparse_features:
features[fea_name] = byte_feature(fea_value)
elif fea_name in dense_features:
features[fea_name] = float_feature(float(label))
tf_example = tf.train.Example(features=tf.train.Features(
feature=features
))
tf_serialized = tf_example.SerializeToString()
return tf_serialized
