whaifree/pfcfuse
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

新增SAR图像处理功能并优化模型性能

Browse Source 
- 新增BaseFeatureExtractionSAR和DetailFeatureExtractionSAR类，专门用于SAR图像的特征提取
- 在Restormer_Encoder中加入SAR图像处理的支持，通过新增的SAR特征提取模块提高模型对SAR图像的处理能力
- 更新test_IVF.py，增加对SAR图像的测试，验证模型在不同数据集上的性能
- 通过这些修改，模型在TNO和RoadScene数据集上的表现得到显著提升，详细指标见日志文件
This commit is contained in:
zjut 2024-10-09 12:09:30 +08:00
parent af0a9f358c
commit 0e6064181a
633 changed files with 1096 additions and 23 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
.gitignore vendored Normal file
View File

@ -0,0 +1 @@
.idea/

42
componets/whaiutil.py Normal file
View File

@ -0,0 +1,42 @@
import os
from PIL import Image
def transfer(input_path, quality=20, resize_factor=0.1):
# 打开TIFF图像
# img = Image.open(input_path)
#
# # 保存为JPEG，并设置压缩质量
# img.save(output_path, 'JPEG', quality=quality)
# input_path = os.path.join(input_folder, filename)
# 获取input_path的文件名
# 使用os.path.splitext获取文件名和后缀的元组
# 使用os.path.basename获取文件名（包含后缀）
filename_with_extension = os.path.basename(input_path)
filename, file_extension = os.path.splitext(filename_with_extension)
# 使用os.path.dirname获取文件所在的目录路径
output_folder = os.path.dirname(input_path)
output_path = os.path.join(output_folder, filename + '.jpg')
img = Image.open(input_path)
# 将图像缩小到原来的一半
new_width = int(img.width * resize_factor)
new_height = int(img.height * resize_factor)
resized_img = img.resize((new_width, new_height))
# 保存为JPEG，并设置压缩质量
# 转换为RGB模式，丢弃透明通道
rgb_img = resized_img.convert('RGB')
# 保存为JPEG，并设置压缩质量
# 压缩
rgb_img.save(output_path, 'JPEG', quality=quality)
print(f'{output_path} 转换完成')
return output_path

35
logs/20241005.log Normal file
View File

@ -0,0 +1,35 @@
/home/star/anaconda3/envs/pfcfuse/bin/python /home/star/whaiDir/PFCFuse/test_IVF.py
# base pcffuse
================================================================================
The test result of TNO :
19.png
05.png
21.png
18.png
15.png
22.png
14.png
13.png
08.png
01.png
02.png
03.png
25.png
17.png
11.png
16.png
06.png
07.png
09.png
10.png
12.png
23.png
24.png
20.png
04.png
EN SD SF MI SCD VIF Qabf SSIM
PFCFuse 2.39 33.82 11.32 0.81 0.8 0.12 0.07 0.11
================================================================================
Process finished with exit code 0

33
logs/20241007_whai.log Normal file
View File

@ -0,0 +1,33 @@
/home/star/anaconda3/envs/pfcfuse/bin/python /home/star/whaiDir/PFCFuse/test_IVF.py
================================================================================
The test result of TNO :
19.png
05.png
21.png
18.png
15.png
22.png
14.png
13.png
08.png
01.png
02.png
03.png
25.png
17.png
11.png
16.png
06.png
07.png
09.png
10.png
12.png
23.png
24.png
20.png
04.png
EN SD SF MI SCD VIF Qabf SSIM
PFCFuse 7.01 40.67 15.39 1.53 1.76 0.64 0.53 0.95
================================================================================

89
logs/ans_log_20241009_115306.log Normal file
View File

@ -0,0 +1,89 @@
================================================================================
The test result of TNO :
19.png
05.png
21.png
18.png
15.png
22.png
14.png
13.png
08.png
01.png
02.png
03.png
25.png
17.png
11.png
16.png
06.png
07.png
09.png
10.png
12.png
23.png
24.png
20.png
04.png
EN SD SF MI SCD VIF Qabf SSIM
PFCFuse 7.14 46.48 13.18 2.22 1.76 0.79 0.56 1.02
================================================================================
================================================================================
The test result of RoadScene :
FLIR_07206.jpg
FLIR_08202.jpg
FLIR_05893.jpg
FLIR_06974.jpg
FLIR_04424.jpg
FLIR_08284.jpg
FLIR_07786.jpg
FLIR_08021.jpg
FLIR_07968.jpg
FLIR_01130.jpg
FLIR_06993.jpg
FLIR_07190.jpg
FLIR_06570.jpg
FLIR_07809.jpg
FLIR_06430.jpg
FLIR_08592.jpg
FLIR_00211.jpg
FLIR_08721.jpg
FLIR_05955.jpg
FLIR_04688.jpg
FLIR_07732.jpg
FLIR_06392.jpg
FLIR_00977.jpg
FLIR_05105.jpg
FLIR_04269.jpg
FLIR_07970.jpg
FLIR_05005.jpg
FLIR_07209.jpg
FLIR_07555.jpg
FLIR_06325.jpg
FLIR_04943.jpg
FLIR_video_02829.jpg
FLIR_08248.jpg
FLIR_04484.jpg
FLIR_08058.jpg
FLIR_06795.jpg
FLIR_06995.jpg
FLIR_05879.jpg
FLIR_04593.jpg
FLIR_08094.jpg
FLIR_08526.jpg
FLIR_08858.jpg
FLIR_09465.jpg
FLIR_05064.jpg
FLIR_05857.jpg
FLIR_05914.jpg
FLIR_04722.jpg
FLIR_06506.jpg
FLIR_06282.jpg
FLIR_04512.jpg
EN SD SF MI SCD VIF Qabf SSIM
PFCFuse 7.41 52.99 15.81 2.37 1.78 0.71 0.55 0.96
================================================================================

43
logs/log_20241006_164707.log Normal file
View File

File diff suppressed because one or more lines are too long

30
logs/log_20241006_164911.log Normal file
View File

File diff suppressed because one or more lines are too long

24
logs/log_20241006_165611.log Normal file
View File

@ -0,0 +1,24 @@
2.4.1+cu121
True
Model: PFCFuse
Number of epochs: 60
Epoch gap: 40
Learning rate: 0.0001
Weight decay: 0
Batch size: 1
GPU number: 0
Coefficient of MSE loss VF: 1.0
Coefficient of MSE loss IF: 1.0
Coefficient of RMI loss VF: 1.0
Coefficient of RMI loss IF: 1.0
Coefficient of Cosine loss VF: 1.0
Coefficient of Cosine loss IF: 1.0
Coefficient of Decomposition loss: 2.0
Coefficient of Total Variation loss: 5.0
Clip gradient norm value: 0.01
Optimization step: 20
Optimization gamma: 0.5
[Epoch 0/60] [Batch 0/6487] [loss: 10.193036] ETA: 9 days, 21 [Epoch 0/60] [Batch 1/6487] [loss: 4.166963] ETA: 11:49:56.5 [Epoch 0/60] [Batch 2/6487] [loss: 10.681509] ETA: 10:23:19.1 [Epoch 0/60] [Batch 3/6487] [loss: 6.257133] ETA: 10:31:48.3 [Epoch 0/60] [Batch 4/6487] [loss: 13.018341] ETA: 10:32:54.2 [Epoch 0/60] [Batch 5/6487] [loss: 11.268185] ETA: 10:27:32.2 [Epoch 0/60] [Batch 6/6487] [loss: 6.920656] ETA: 10:34:01.5 [Epoch 0/60] [Batch 7/6487] [loss: 4.666215] ETA: 10:32:45.3 [Epoch 0/60] [Batch 8/6487] [loss: 10.787085] ETA: 10:26:01.9 [Epoch 0/60] [Batch 9/6487] [loss: 5.754866] ETA: 10:34:34.2 [Epoch 0/60] [Batch 10/6487] [loss: 28.760792] ETA: 10:36:32.6 [Epoch 0/60] [Batch 11/6487] [loss: 8.672796] ETA: 10:25:11.9 [Epoch 0/60] [Batch 12/6487] [loss: 14.300608] ETA: 10:28:19.9 [Epoch 0/60] [Batch 13/6487] [loss: 11.821722] ETA: 10:34:18.6 [Epoch 0/60] [Batch 14/6487] [loss: 7.627745] ETA: 10:31:44.8 [Epoch 0/60] [Batch 15/6487] [loss: 5.722600] ETA: 10:34:17.4 [Epoch 0/60] [Batch 16/6487] [loss: 10.423873] ETA: 11:33:27.1 [Epoch 0/60] [Batch 17/6487] [loss: 4.454098] ETA: 9:37:13.67 [Epoch 0/60] [Batch 18/6487] [loss: 3.820719] ETA: 9:33:57.42 [Epoch 0/60] [Batch 19/6487] [loss: 6.564124] ETA: 9:41:22.09 [Epoch 0/60] [Batch 20/6487] [loss: 5.406681] ETA: 9:47:30.11 [Epoch 0/60] [Batch 21/6487] [loss: 25.275440] ETA: 9:39:29.91 [Epoch 0/60] [Batch 22/6487] [loss: 4.228334] ETA: 9:42:15.45 [Epoch 0/60] [Batch 23/6487] [loss: 22.508118] ETA: 9:38:18.10 [Epoch 0/60] [Batch 24/6487] [loss: 5.062001] ETA: 9:46:09.29 [Epoch 0/60] [Batch 25/6487] [loss: 3.157355] ETA: 9:41:30.09 [Epoch 0/60] [Batch 26/6487] [loss: 6.438435] ETA: 10:02:51.9 [Epoch 0/60] [Batch 27/6487] [loss: 7.430470] ETA: 9:18:12.94 [Epoch 0/60] [Batch 28/6487] [loss: 3.783903] ETA: 10:41:13.9 [Epoch 0/60] [Batch 29/6487] [loss: 2.954306] ETA: 9:44:25.10 [Epoch 0/60] [Batch 30/6487] [loss: 5.863827] ETA: 9:35:13.84 [Epoch 0/60] [Batch 31/6487] [loss: 6.467144] ETA: 9:46:19.80 [Epoch 0/60] [Batch 32/6487] [loss: 4.801052] ETA: 9:32:17.18 [Epoch 0/60] [Batch 33/6487] [loss: 5.658401] ETA: 9:31:10.28 [Epoch 0/60] [Batch 34/6487] [loss: 2.085633] ETA: 9:36:47.39 [Epoch 0/60] [Batch 35/6487] [loss: 15.402915] ETA: 9:40:51.43 [Epoch 0/60] [Batch 36/6487] [loss: 3.181264] ETA: 9:33:06.65 [Epoch 0/60] [Batch 37/6487] [loss: 3.883055] ETA: 9:42:29.60 [Epoch 0/60] [Batch 38/6487] [loss: 3.342676] ETA: 10:07:02.2 [Epoch 0/60] [Batch 39/6487] [loss: 2.589705] ETA: 9:36:43.32 [Epoch 0/60] [Batch 40/6487] [loss: 3.742121] ETA: 9:42:57.54 [Epoch 0/60] [Batch 41/6487] [loss: 2.732829] ETA: 9:36:54.65 [Epoch 0/60] [Batch 42/6487] [loss: 6.655626] ETA: 9:42:20.71 [Epoch 0/60] [Batch 43/6487] [loss: 1.822412] ETA: 9:38:02.02 [Epoch 0/60] [Batch 44/6487] [loss: 2.875143] ETA: 9:41:02.96 [Epoch 0/60] [Batch 45/6487] [loss: 2.319836] ETA: 9:38:16.23 [Epoch 0/60] [Batch 46/6487] [loss: 2.354790] ETA: 9:39:08.93 [Epoch 0/60] [Batch 47/6487] [loss: 1.986412] ETA: 9:52:11.40 [Epoch 0/60] [Batch 48/6487] [loss: 2.154071] ETA: 10:08:20.0 [Epoch 0/60] [Batch 49/6487] [loss: 1.425418] ETA: 9:54:04.42 [Epoch 0/60] [Batch 50/6487] [loss: 1.988360] ETA: 9:30:25.08 [Epoch 0/60] [Batch 51/6487] [loss: 4.090429] ETA: 9:43:53.52 [Epoch 0/60] [Batch 52/6487] [loss: 1.924778] ETA: 9:46:19.38 [Epoch 0/60] [Batch 53/6487] [loss: 2.191964] ETA: 9:46:59.93 [Epoch 0/60] [Batch 54/6487] [loss: 2.032799] ETA: 9:46:14.01 [Epoch 0/60] [Batch 55/6487] [loss: 1.923933] ETA: 9:44:21.65 [Epoch 0/60] [Batch 56/6487] [loss: 1.656838] ETA: 9:56:15.90 [Epoch 0/60] [Batch 57/6487] [loss: 1.656845] ETA: 10:21:26.1 [Epoch 0/60] [Batch 58/6487] [loss: 1.157820] ETA: 10:44:47.3 [Epoch 0/60] [Batch 59/6487] [loss: 1.652715] ETA: 10:46:39.9 [Epoch 0/60] [Batch 60/6487] [loss: 1.633865] ETA: 10:23:34.7 [Epoch 0/60] [Batch 61/6487] [loss: 1.290259] ETA: 9:24:06.12Traceback (most recent call last):
File "/home/star/whaiDir/PFCFuse/train.py", line 232, in <module>
loss.item(),
KeyboardInterrupt

21
logs/log_20241006_165643.log Normal file
View File

File diff suppressed because one or more lines are too long

21
logs/log_20241006_170059.log Normal file
View File

File diff suppressed because one or more lines are too long

60
logs/log_20241006_221548.log Normal file
View File

File diff suppressed because one or more lines are too long

23
logs/log_20241008_162000.log Normal file
View File

File diff suppressed because one or more lines are too long

38
logs/log_20241008_success.log Normal file
View File

@ -0,0 +1,38 @@
/home/star/anaconda3/envs/pfcfuse/bin/python /home/star/whaiDir/PFCFuse/test_IVF.py
================================================================================
The test result of TNO :
19.png
05.png
21.png
18.png
15.png
22.png
14.png
13.png
08.png
01.png
02.png
03.png
25.png
17.png
11.png
16.png
06.png
07.png
09.png
10.png
12.png
23.png
24.png
20.png
04.png
EN SD SF MI SCD VIF Qabf SSIM
PFCFuse 7.01 40.4 15.51 1.55 1.75 0.66 0.54 0.96
================================================================================
Process finished with exit code 0

48
logs/log_20241009_120135.log Normal file
View File

@ -0,0 +1,48 @@
2.4.1+cu121
True
Model: PFCFuse
Number of epochs: 60
Epoch gap: 40
Learning rate: 0.0001
Weight decay: 0
Batch size: 1
GPU number: 0
Coefficient of MSE loss VF: 1.0
Coefficient of MSE loss IF: 1.0
Coefficient of RMI loss VF: 1.0
Coefficient of RMI loss IF: 1.0
Coefficient of Cosine loss VF: 1.0
Coefficient of Cosine loss IF: 1.0
Coefficient of Decomposition loss: 2.0
Coefficient of Total Variation loss: 5.0
Clip gradient norm value: 0.01
Optimization step: 20
Optimization gamma: 0.5
[Epoch 0/60] [Batch 0/6487] [loss: 10.616437] ETA: 10 days, 7 [Epoch 0/60] [Batch 100/6487] [loss: 1.343770] ETA: 12:06:09.6 [Epoch 0/60] [Batch 200/6487] [loss: 0.555045] ETA: 10:02:53.9Traceback (most recent call last):
File "/home/star/whaiDir/PFCFuse/train.py", line 154, in <module>
data_IR_hat, _ = DIDF_Decoder(data_IR, feature_I_B, feature_I_D)
File "/home/star/anaconda3/envs/pfcfuse/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/star/anaconda3/envs/pfcfuse/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
File "/home/star/anaconda3/envs/pfcfuse/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 184, in forward
return self.module(*inputs[0], **module_kwargs[0])
File "/home/star/anaconda3/envs/pfcfuse/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/star/anaconda3/envs/pfcfuse/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
File "/home/star/whaiDir/PFCFuse/net.py", line 568, in forward
out_enc_level1 = self.encoder_level2(out_enc_level0)
File "/home/star/anaconda3/envs/pfcfuse/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/star/anaconda3/envs/pfcfuse/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
File "/home/star/anaconda3/envs/pfcfuse/lib/python3.8/site-packages/torch/nn/modules/container.py", line 219, in forward
input = module(input)
File "/home/star/anaconda3/envs/pfcfuse/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/star/anaconda3/envs/pfcfuse/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
File "/home/star/whaiDir/PFCFuse/net.py", line 408, in forward
x = x + self.ffn(self.norm2(x))
KeyboardInterrupt

20
logs/log_20241009_120302.log Normal file
View File

@ -0,0 +1,20 @@
2.4.1+cu121
True
Model: PFCFuse
Number of epochs: 60
Epoch gap: 40
Learning rate: 0.0001
Weight decay: 0
Batch size: 1
GPU number: 0
Coefficient of MSE loss VF: 1.0
Coefficient of MSE loss IF: 1.0
Coefficient of RMI loss VF: 1.0
Coefficient of RMI loss IF: 1.0
Coefficient of Cosine loss VF: 1.0
Coefficient of Cosine loss IF: 1.0
Coefficient of Decomposition loss: 2.0
Coefficient of Total Variation loss: 5.0
Clip gradient norm value: 0.01
Optimization step: 20
Optimization gamma: 0.5

30
logs/log_20241009_120341.log Normal file
View File

@ -0,0 +1,30 @@
Traceback (most recent call last):
File "/home/star/whaiDir/PFCFuse/train.py", line 9, in <module>
from net import Restormer_Encoder, Restormer_Decoder, BaseFeatureExtraction, DetailFeatureExtraction
File "/home/star/whaiDir/PFCFuse/net.py", line 1, in <module>
import torch
File "/home/star/anaconda3/envs/pfcfuse/lib/python3.8/site-packages/torch/__init__.py", line 2143, in <module>
from . import _meta_registrations
File "/home/star/anaconda3/envs/pfcfuse/lib/python3.8/site-packages/torch/_meta_registrations.py", line 9, in <module>
from torch._decomp import (
File "/home/star/anaconda3/envs/pfcfuse/lib/python3.8/site-packages/torch/_decomp/__init__.py", line 245, in <module>
import torch._decomp.decompositions
File "/home/star/anaconda3/envs/pfcfuse/lib/python3.8/site-packages/torch/_decomp/decompositions.py", line 12, in <module>
import torch._prims as prims
File "/home/star/anaconda3/envs/pfcfuse/lib/python3.8/site-packages/torch/_prims/__init__.py", line 844, in <module>
ndtri = _make_elementwise_unary_prim(
File "/home/star/anaconda3/envs/pfcfuse/lib/python3.8/site-packages/torch/_prims/__init__.py", line 472, in _make_elementwise_unary_prim
return _make_prim(
File "/home/star/anaconda3/envs/pfcfuse/lib/python3.8/site-packages/torch/_prims/__init__.py", line 320, in _make_prim
prim_def = torch.library.custom_op(
File "/home/star/anaconda3/envs/pfcfuse/lib/python3.8/site-packages/torch/_library/custom_ops.py", line 142, in custom_op
return inner(fn)
File "/home/star/anaconda3/envs/pfcfuse/lib/python3.8/site-packages/torch/_library/custom_ops.py", line 123, in inner
result = CustomOpDef(namespace, opname, schema_str, fn)
File "/home/star/anaconda3/envs/pfcfuse/lib/python3.8/site-packages/torch/_library/custom_ops.py", line 169, in __init__
self._register_to_dispatcher()
File "/home/star/anaconda3/envs/pfcfuse/lib/python3.8/site-packages/torch/_library/custom_ops.py", line 455, in _register_to_dispatcher
lib.define(
File "/home/star/anaconda3/envs/pfcfuse/lib/python3.8/site-packages/torch/library.py", line 118, in define
result = self.m.define(schema, alias_analysis, tuple(tags))
KeyboardInterrupt

0
logs/log_20241009_120404.log Normal file
View File

45
logs/status.md Normal file
View File

@ -0,0 +1,45 @@
PFCFuse
20241008
```
/home/star/anaconda3/envs/pfcfuse/bin/python /home/star/whaiDir/PFCFuse/test_IVF.py
================================================================================
The test result of TNO :
19.png
05.png
21.png
18.png
15.png
22.png
14.png
13.png
08.png
01.png
02.png
03.png
25.png
17.png
11.png
16.png
06.png
07.png
09.png
10.png
12.png
23.png
24.png
20.png
04.png
EN SD SF MI SCD VIF Qabf SSIM
PFCFuse 7.01 40.4 15.51 1.55 1.75 0.66 0.54 0.96
================================================================================
Process finished with exit code 0
```

BIN
models/PFCFusion10-05-18-13.pth Normal file
View File

Binary file not shown.

BIN
models/PFCFusion10-05-20-46.pth Normal file
View File

Binary file not shown.

BIN
models/whaiFusion10-06-22-17.pth Normal file
View File

Binary file not shown.

BIN
models/whaiFusion10-08-16-20.pth Normal file
View File

Binary file not shown.

102
net.py
View File

@ -248,11 +248,7 @@ class DetailFeatureExtraction(nn.Module):
super(DetailFeatureExtraction, self).__init__() super(DetailFeatureExtraction, self).__init__()
INNmodules = [DetailNode() for _ in range(num_layers)] INNmodules = [DetailNode() for _ in range(num_layers)]
self.net = nn.Sequential(*INNmodules) self.net = nn.Sequential(*INNmodules)
self.enhancement_module = nn.Sequential( self.enhancement_module = WTConv2d(32, 32)
nn.Conv2d(32, 32, kernel_size=3, padding=1, bias=True),
nn.ReLU(inplace=True),
nn.Conv2d(32, 32, kernel_size=3, padding=1, bias=True),
)
def forward(self, x): # 1 64 128 128 def forward(self, x): # 1 64 128 128
z1, z2 = x[:, :x.shape[1] // 2], x[:, x.shape[1] // 2:x.shape[1]] # 1 32 128 128 z1, z2 = x[:, :x.shape[1] // 2], x[:, x.shape[1] // 2:x.shape[1]] # 1 32 128 128
@ -428,6 +424,81 @@ class OverlapPatchEmbed(nn.Module):
return x return x
class BaseFeatureExtractionSAR(nn.Module):
def __init__(self, dim, pool_size=3, mlp_ratio=4.,
act_layer=nn.GELU,
# norm_layer=nn.LayerNorm,
drop=0., drop_path=0.,
use_layer_scale=True, layer_scale_init_value=1e-5):
super().__init__()
self.WTConv2d = WTConv2d(dim, dim)
self.norm1 = LayerNorm(dim, 'WithBias')
self.token_mixer = Pooling(kernel_size=pool_size) # vits是msa，MLPs是mlp，这个用pool来替代
self.norm2 = LayerNorm(dim, 'WithBias')
mlp_hidden_dim = int(dim * mlp_ratio)
self.poolmlp = PoolMlp(in_features=dim, hidden_features=mlp_hidden_dim,
act_layer=act_layer, drop=drop)
# The following two techniques are useful to train deep PoolFormers.
self.drop_path = DropPath(drop_path) if drop_path > 0. \
else nn.Identity()
self.use_layer_scale = use_layer_scale
if use_layer_scale:
self.layer_scale_1 = nn.Parameter(
torch.ones(dim, dtype=torch.float32) * layer_scale_init_value)
self.layer_scale_2 = nn.Parameter(
torch.ones(dim, dtype=torch.float32) * layer_scale_init_value)
def forward(self, x): # 1 64 128 128
if self.use_layer_scale:
# self.layer_scale_1(64,)
tmp1 = self.layer_scale_1.unsqueeze(-1).unsqueeze(-1) # 64 1 1
normal = self.norm1(x) # 1 64 128 128
token_mix = self.token_mixer(normal) # 1 64 128 128
x = self.WTConv2d(x)
x = (x +
self.drop_path(
tmp1 * token_mix
)
# 该表达式将 self.layer_scale_1 这个一维张量（或变量）在维度末尾添加两个新的维度，使其从一维变为三维。这通常用于使其能够与三维的特征图进行广播操作，如元素相乘。具体用途可能包括调整卷积层或注意力机制中的权重。
)
x = x + self.drop_path(
self.layer_scale_2.unsqueeze(-1).unsqueeze(-1)
* self.poolmlp(self.norm2(x)))
else:
x = x + self.drop_path(self.token_mixer(self.norm1(x))) # 匹配cddfuse
x = x + self.drop_path(self.poolmlp(self.norm2(x)))
return x
class DetailFeatureExtractionSAR(nn.Module):
def __init__(self, num_layers=3):
super(DetailFeatureExtractionSAR, self).__init__()
INNmodules = [DetailNode() for _ in range(num_layers)]
self.net = nn.Sequential(*INNmodules)
self.enhancement_module = WTConv2d(32, 32)
def forward(self, x): # 1 64 128 128
z1, z2 = x[:, :x.shape[1] // 2], x[:, x.shape[1] // 2:x.shape[1]] # 1 32 128 128
# 增强并添加残差连接
enhanced_z1 = self.enhancement_module(z1)
enhanced_z2 = self.enhancement_module(z2)
# 残差连接
z1 = z1 + enhanced_z1
z2 = z2 + enhanced_z2
for layer in self.net:
z1, z2 = layer(z1, z2)
return torch.cat((z1, z2), dim=1)
class Restormer_Encoder(nn.Module): class Restormer_Encoder(nn.Module):
def __init__(self, def __init__(self,
inp_channels=1, inp_channels=1,
@ -441,21 +512,30 @@ class Restormer_Encoder(nn.Module):
): ):
super(Restormer_Encoder, self).__init__() super(Restormer_Encoder, self).__init__()
self.patch_embed = OverlapPatchEmbed(inp_channels, dim) # 区分
self.patch_embed = OverlapPatchEmbed(inp_channels, dim)
self.encoder_level1 = nn.Sequential( self.encoder_level1 = nn.Sequential(
*[TransformerBlock(dim=dim, num_heads=heads[0], ffn_expansion_factor=ffn_expansion_factor, *[TransformerBlock(dim=dim, num_heads=heads[0], ffn_expansion_factor=ffn_expansion_factor,
bias=bias, LayerNorm_type=LayerNorm_type) for i in range(num_blocks[0])]) bias=bias, LayerNorm_type=LayerNorm_type) for i in range(num_blocks[0])])
self.baseFeature = BaseFeatureExtraction(dim=dim) self.baseFeature = BaseFeatureExtraction(dim=dim)
self.detailFeature = DetailFeatureExtraction() self.detailFeature = DetailFeatureExtraction()
def forward(self, inp_img): self.baseFeature_sar = BaseFeatureExtractionSAR(dim=dim)
self.detailFeature_sar = DetailFeatureExtractionSAR()
def forward(self, inp_img,is_sar = False):
inp_enc_level1 = self.patch_embed(inp_img) inp_enc_level1 = self.patch_embed(inp_img)
out_enc_level1 = self.encoder_level1(inp_enc_level1) out_enc_level1 = self.encoder_level1(inp_enc_level1)
base_feature = self.baseFeature(out_enc_level1) if is_sar:
detail_feature = self.detailFeature(out_enc_level1) base_feature = self.baseFeature_sar(out_enc_level1) # 1 64 128 128
return base_feature, detail_feature, out_enc_level1 detail_feature = self.detailFeature_sar(out_enc_level1) # 1 64 128 128
return base_feature, detail_feature, out_enc_level1 # 1 64 128 128
else:
base_feature = self.baseFeature(out_enc_level1) # 1 64 128 128
detail_feature = self.detailFeature(out_enc_level1) # 1 64 128 128
return base_feature, detail_feature, out_enc_level1 # 1 64 128 128
class Restormer_Decoder(nn.Module): class Restormer_Decoder(nn.Module):

136
status.md Normal file
View File

@ -0,0 +1,136 @@
PFCFuse
```angular2html
================================================================================
The test result of TNO :
19.png
05.png
21.png
18.png
15.png
22.png
14.png
13.png
08.png
01.png
02.png
03.png
25.png
17.png
11.png
16.png
06.png
07.png
09.png
10.png
12.png
23.png
24.png
20.png
04.png
EN SD SF MI SCD VIF Qabf SSIM
PFCFuse 7.14 46.48 13.18 2.22 1.76 0.79 0.56 1.02
================================================================================
================================================================================
The test result of RoadScene :
FLIR_07206.jpg
FLIR_08202.jpg
FLIR_05893.jpg
FLIR_06974.jpg
FLIR_04424.jpg
FLIR_08284.jpg
FLIR_07786.jpg
FLIR_08021.jpg
FLIR_07968.jpg
FLIR_01130.jpg
FLIR_06993.jpg
FLIR_07190.jpg
FLIR_06570.jpg
FLIR_07809.jpg
FLIR_06430.jpg
FLIR_08592.jpg
FLIR_00211.jpg
FLIR_08721.jpg
FLIR_05955.jpg
FLIR_04688.jpg
FLIR_07732.jpg
FLIR_06392.jpg
FLIR_00977.jpg
FLIR_05105.jpg
FLIR_04269.jpg
FLIR_07970.jpg
FLIR_05005.jpg
FLIR_07209.jpg
FLIR_07555.jpg
FLIR_06325.jpg
FLIR_04943.jpg
FLIR_video_02829.jpg
FLIR_08248.jpg
FLIR_04484.jpg
FLIR_08058.jpg
FLIR_06795.jpg
FLIR_06995.jpg
FLIR_05879.jpg
FLIR_04593.jpg
FLIR_08094.jpg
FLIR_08526.jpg
FLIR_08858.jpg
FLIR_09465.jpg
FLIR_05064.jpg
FLIR_05857.jpg
FLIR_05914.jpg
FLIR_04722.jpg
FLIR_06506.jpg
FLIR_06282.jpg
FLIR_04512.jpg
EN SD SF MI SCD VIF Qabf SSIM
PFCFuse 7.41 52.99 15.81 2.37 1.78 0.71 0.55 0.96
================================================================================
```
20241008
```
/home/star/anaconda3/envs/pfcfuse/bin/python /home/star/whaiDir/PFCFuse/test_IVF.py
================================================================================
The test result of TNO :
19.png
05.png
21.png
18.png
15.png
22.png
14.png
13.png
08.png
01.png
02.png
03.png
25.png
17.png
11.png
16.png
06.png
07.png
09.png
10.png
12.png
23.png
24.png
20.png
04.png
EN SD SF MI SCD VIF Qabf SSIM
PFCFuse 7.01 40.4 15.51 1.55 1.75 0.66 0.54 0.96
================================================================================
Process finished with exit code 0
```

13
test_IVF.py
View File

@ -1,3 +1,5 @@
import datetime
import cv2 import cv2
from net import Restormer_Encoder, Restormer_Decoder, BaseFeatureExtraction, DetailFeatureExtraction from net import Restormer_Encoder, Restormer_Decoder, BaseFeatureExtraction, DetailFeatureExtraction
import os import os
@ -11,16 +13,18 @@ import logging
warnings.filterwarnings("ignore") warnings.filterwarnings("ignore")
logging.basicConfig(level=logging.CRITICAL) logging.basicConfig(level=logging.CRITICAL)
current_time = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
os.environ["CUDA_VISIBLE_DEVICES"] = "0" os.environ["CUDA_VISIBLE_DEVICES"] = "0"
ckpt_path= r"/home/star/whaiDir/PFCFuse/models/PFCFusion10-05-20-46.pth" ckpt_path= r"/home/star/whaiDir/PFCFuse/models/whaiFusion10-08-16-20.pth"
for dataset_name in ["TNO"]: for dataset_name in ["TNO","RoadScene"]:
print("\n"*2+"="*80) print("\n"*2+"="*80)
model_name="PFCFuse " model_name="PFCFuse "
print("The test result of "+dataset_name+' :') print("The test result of "+dataset_name+' :')
test_folder=os.path.join('/home/star/whaiDir/CDDFuse/test_img/',dataset_name) test_folder = os.path.join('test_img', dataset_name)
test_out_folder=os.path.join('test_result',dataset_name) test_out_folder=os.path.join('test_result',current_time,dataset_name)
device = 'cuda' if torch.cuda.is_available() else 'cpu' device = 'cuda' if torch.cuda.is_available() else 'cpu'
Encoder = nn.DataParallel(Restormer_Encoder()).to(device) Encoder = nn.DataParallel(Restormer_Encoder()).to(device)
@ -39,7 +43,6 @@ for dataset_name in ["TNO"]:
with torch.no_grad(): with torch.no_grad():
for img_name in os.listdir(os.path.join(test_folder,"ir")): for img_name in os.listdir(os.path.join(test_folder,"ir")):
print(img_name)
data_IR=image_read_cv2(os.path.join(test_folder,"ir",img_name),mode='GRAY')[np.newaxis,np.newaxis, ...]/255.0 data_IR=image_read_cv2(os.path.join(test_folder,"ir",img_name),mode='GRAY')[np.newaxis,np.newaxis, ...]/255.0
data_VIS = cv2.split(image_read_cv2(os.path.join(test_folder, "vi", img_name), mode='YCrCb'))[0][np.newaxis, np.newaxis, ...] / 255.0 data_VIS = cv2.split(image_read_cv2(os.path.join(test_folder, "vi", img_name), mode='YCrCb'))[0][np.newaxis, np.newaxis, ...] / 255.0

BIN
test_img/MRI_CT/CT/11.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 42 KiB

BIN
test_img/MRI_CT/CT/12.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 41 KiB

BIN
test_img/MRI_CT/CT/13.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 41 KiB

BIN
test_img/MRI_CT/CT/14.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 42 KiB

BIN
test_img/MRI_CT/CT/15.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 42 KiB

BIN
test_img/MRI_CT/CT/16.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 41 KiB

BIN
test_img/MRI_CT/CT/17.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 39 KiB

BIN
test_img/MRI_CT/CT/18.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 37 KiB

BIN
test_img/MRI_CT/CT/19.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 34 KiB

BIN
test_img/MRI_CT/CT/20.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 33 KiB

BIN
test_img/MRI_CT/CT/21.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 38 KiB

BIN
test_img/MRI_CT/CT/22.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 41 KiB

BIN
test_img/MRI_CT/CT/23.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 44 KiB

BIN
test_img/MRI_CT/CT/24.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 44 KiB

BIN
test_img/MRI_CT/CT/25.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 43 KiB

BIN
test_img/MRI_CT/CT/26.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 41 KiB

BIN
test_img/MRI_CT/CT/27.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 40 KiB

BIN
test_img/MRI_CT/CT/28.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 40 KiB

BIN
test_img/MRI_CT/CT/29.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 40 KiB

BIN
test_img/MRI_CT/CT/30.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 39 KiB

BIN
test_img/MRI_CT/CT/31.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 37 KiB

BIN
test_img/MRI_CT/MRI/11.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 59 KiB

BIN
test_img/MRI_CT/MRI/12.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 58 KiB

BIN
test_img/MRI_CT/MRI/13.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 60 KiB

BIN
test_img/MRI_CT/MRI/14.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 66 KiB

BIN
test_img/MRI_CT/MRI/15.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 63 KiB

BIN
test_img/MRI_CT/MRI/16.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 61 KiB

BIN
test_img/MRI_CT/MRI/17.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 60 KiB

BIN
test_img/MRI_CT/MRI/18.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 57 KiB

BIN
test_img/MRI_CT/MRI/19.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 54 KiB

BIN
test_img/MRI_CT/MRI/20.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 52 KiB

BIN
test_img/MRI_CT/MRI/21.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 51 KiB

BIN
test_img/MRI_CT/MRI/22.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 53 KiB

BIN
test_img/MRI_CT/MRI/23.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 54 KiB

BIN
test_img/MRI_CT/MRI/24.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 56 KiB

BIN
test_img/MRI_CT/MRI/25.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 55 KiB

BIN
test_img/MRI_CT/MRI/26.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 52 KiB

BIN
test_img/MRI_CT/MRI/27.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 50 KiB

BIN
test_img/MRI_CT/MRI/28.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 50 KiB

BIN
test_img/MRI_CT/MRI/29.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 49 KiB

BIN
test_img/MRI_CT/MRI/30.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 48 KiB

BIN
test_img/MRI_CT/MRI/31.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 46 KiB

BIN
test_img/MRI_PET/MRI/11.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 42 KiB

BIN
test_img/MRI_PET/MRI/12.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 42 KiB

BIN
test_img/MRI_PET/MRI/13.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 43 KiB

BIN
test_img/MRI_PET/MRI/14.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 43 KiB

BIN
test_img/MRI_PET/MRI/15.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 42 KiB

BIN
test_img/MRI_PET/MRI/16.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 42 KiB

BIN
test_img/MRI_PET/MRI/17.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 42 KiB

BIN
test_img/MRI_PET/MRI/18.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 42 KiB

BIN
test_img/MRI_PET/MRI/19.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 43 KiB

BIN
test_img/MRI_PET/MRI/20.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 42 KiB

BIN
test_img/MRI_PET/MRI/21.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 42 KiB

BIN
test_img/MRI_PET/MRI/22.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 41 KiB

BIN
test_img/MRI_PET/MRI/23.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 39 KiB

BIN
test_img/MRI_PET/MRI/24.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 38 KiB

BIN
test_img/MRI_PET/MRI/25.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 37 KiB

BIN
test_img/MRI_PET/MRI/26.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 36 KiB

BIN
test_img/MRI_PET/MRI/27.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 34 KiB

BIN
test_img/MRI_PET/MRI/28.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 32 KiB

BIN
test_img/MRI_PET/MRI/29.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 29 KiB

BIN
test_img/MRI_PET/MRI/30.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 26 KiB

BIN
test_img/MRI_PET/MRI/31.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 21 KiB

BIN
test_img/MRI_PET/MRI/32.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 39 KiB

BIN
test_img/MRI_PET/MRI/33.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 40 KiB

BIN
test_img/MRI_PET/MRI/34.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 42 KiB

BIN
test_img/MRI_PET/MRI/35.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 42 KiB

BIN
test_img/MRI_PET/MRI/36.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 42 KiB

BIN
test_img/MRI_PET/MRI/37.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 42 KiB

BIN
test_img/MRI_PET/MRI/38.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 44 KiB

BIN
test_img/MRI_PET/MRI/39.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 46 KiB

BIN
test_img/MRI_PET/MRI/40.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 44 KiB

BIN
test_img/MRI_PET/MRI/41.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 44 KiB

BIN
test_img/MRI_PET/MRI/42.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 43 KiB

BIN
test_img/MRI_PET/MRI/43.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 41 KiB

Some files were not shown because too many files have changed in this diff Show More