Overview
Submit package home:Aloysius:branches:Essentials / x265 to package Essentials / x265
x265.changes
Changed
x
1
2
-------------------------------------------------------------------3
+Tue Oct 9 20:03:53 UTC 2018 - aloisio@gmx.com4
+5
+- Update to version 2.96
+ New features:7
+ * Support for chunked encoding8
+ + :option:`--chunk-start and --chunk-end`9
+ + Frames preceding first frame of chunk in display order10
+ will be encoded, however, they will be discarded in the11
+ bitstream.12
+ + Frames following last frame of the chunk in display order13
+ will be used in taking lookahead decisions, but, they will14
+ not be encoded.15
+ + This feature can be enabled only in closed GOP structures.16
+ Default disabled.17
+ * Support for HDR10+ version 1 SEI messages.18
+ Encoder enhancements:19
+ * Create API function for allocating and freeing20
+ x265_analysis_data.21
+ * CEA 608/708 support: Read SEI messages from text file and22
+ encode it using userSEI message.23
+ Bug fixes:24
+ * Disable noise reduction when vbv is enabled.25
+ * Support minLuma and maxLuma values changed by the26
+ commandline.27
+ version 2.828
+ New features:29
+ * :option:`--asm avx512` used to enable AVX-512 in x265.30
+ Default disabled. 31
+ + For 4K main10 high-quality encoding, we are seeing good32
+ gains; for other resolutions and presets, we don't33
+ recommend using this setting for now.34
+ * :option:`--dynamic-refine` dynamically switches between35
+ different inter refine levels. Default disabled.36
+ + It is recommended to use :option:`--refine-intra 4' with37
+ dynamic refinement for a better trade-off between encode38
+ efficiency and performance than using static refinement.39
+ * :option:`--single-sei`40
+ + Encode SEI messages in a single NAL unit instead of41
+ multiple NAL units. Default disabled.42
+ * :option:`--max-ausize-factor` controls the maximum AU size43
+ defined in HEVC specification.44
+ + It represents the percentage of maximum AU size used.45
+ Default is 1.46
+ * VMAF (Video Multi-Method Assessment Fusion)47
+ + Added VMAF support for objective quality measurement of a48
+ video sequence.49
+ + Enable cmake option ENABLE_LIBVMAF to report per frame and50
+ aggregate VMAF score. The frame level VMAF score does not51
+ include temporal scores.52
+ + This is supported only on linux for now.53
+ Encoder enhancements:54
+ * Introduced refine-intra level 4 to improve quality.55
+ * Support for HLG-graded content and pic_struct in SEI message.56
+ Bug Fixes:57
+ * Fix 32 bit build error (using CMAKE GUI) in Linux.58
+ * Fix 32 bit build error for asm primitives.59
+ * Fix build error on mac OS.60
+ * Fix VBV Lookahead in analysis load to achieve target bitrate.61
+62
+- Added x265-fix_enable512.patch63
+64
+-------------------------------------------------------------------65
Fri May 4 22:21:57 UTC 2018 - zaitor@opensuse.org66
67
- Build with nasm >= 2.13 for openSUSE Leap 42.3 and SLE-12, since68
x265.spec
Changed
83
1
2
# based on the spec file from https://build.opensuse.org/package/view_file/home:Simmphonie/libx265/3
4
Name: x2655
-%define soname 1516
+%define soname 1657
%define libname lib%{name}8
%define libsoname %{libname}-%{soname}9
-Version: 2.710
+Version: 2.911
Release: 012
License: GPL-2.0+13
Summary: A free h265/HEVC encoder - encoder binary14
15
Source0: https://bitbucket.org/multicoreware/x265/downloads/%{name}_%{version}.tar.gz16
Patch0: arm.patch17
Patch1: x265.pkgconfig.patch18
+Patch2: x265-fix_enable512.patch19
BuildRequires: gcc20
BuildRequires: gcc-c++21
BuildRequires: cmake >= 2.8.822
BuildRequires: pkg-config23
BuildRequires: nasm >= 2.1324
-%if 0%{?suse_version} > 131025
%ifarch x86_6426
BuildRequires: libnuma-devel >= 2.0.927
%endif28
-%endif29
-BuildRoot: %{_tmppath}/%{name}-%{version}-build30
31
%description32
x265 is a free library for encoding next-generation H265/HEVC video33
34
35
%description -n %{libname}-devel36
x265 is a free library for encoding next-generation H265/HEVC video37
-streams. 38
+streams.39
40
%prep41
%setup -q -n %{name}_%{version}42
%patch0 -p143
%patch1 -p144
+%patch2 -p145
46
sed -i -e "s/0.0/%{soname}.0/g" source/cmake/version.cmake47
48
49
%build50
-%if 0%{?suse_version} < 133051
+%if 0%{?suse_version} < 150052
cd source53
%else54
%define __builddir ./source/build55
56
make %{?_smp_mflags}57
58
%install59
-%if 0%{?suse_version} < 133060
+%if 0%{?suse_version} < 150061
cd source62
%endif63
%cmake_install64
65
%postun -n %{libsoname} -p /sbin/ldconfig66
67
%files -n %{libsoname}68
-%defattr(0644,root,root)69
%{_libdir}/%{libname}.so.%{soname}*70
71
-%files 72
-%defattr(0755,root,root)73
+%files74
%{_bindir}/%{name}75
76
%files -n %{libname}-devel77
-%defattr(0644,root,root)78
+%license COPYING79
+%doc readme.rst80
%{_includedir}/%{name}.h81
%{_includedir}/%{name}_config.h82
%{_libdir}/pkgconfig/%{name}.pc83
x265-fix_enable512.patch
Added
27
1
2
+--- a/source/common/cpu.cpp3
++++ b/source/common/cpu.cpp4
+@@ -110,6 +110,11 @@ const cpu_name_t cpu_names[] =5
+ { "", 0 },6
+ };7
+ 8
++bool detect512()9
++{10
++ return(enable512);11
++}12
++13
+ #if X265_ARCH_X8614
+ 15
+ extern "C" {16
+@@ -123,10 +128,6 @@ uint64_t PFX(cpu_xgetbv)(int xcr);17
+ #pragma warning(disable: 4309) // truncation of constant value18
+ #endif19
+ 20
+-bool detect512()21
+-{22
+- return(enable512);23
+-}24
+ uint32_t cpu_detect(bool benableavx512 )25
+ {26
+ 27
x265_2.7.tar.gz/.hg_archival.txt -> x265_2.9.tar.gz/.hg_archival.txt
Changed
8
1
2
repo: 09fe40627f03a0f9c3e6ac78b22ac93da23f9fdf3
-node: e41a9bf2bac4a7af2bec2bbadf91e63752d320ef4
+node: f9681d731f2e56c2ca185cec10daece5939bee075
branch: stable6
-tag: 2.77
+tag: 2.98
x265_2.7.tar.gz/.hgtags -> x265_2.9.tar.gz/.hgtags
Changed
7
1
2
e7a4dd48293b7956d4a20df257d23904cc78e376 2.43
64b2d0bf45a52511e57a6b7299160b961ca3d51c 2.54
0e9ea76945c89962cd46cee6537586e2054b2935 2.65
+e41a9bf2bac4a7af2bec2bbadf91e63752d320ef 2.76
+a158a3a029663133455268e2a63ae6b0af2df720 2.87
x265_2.7.tar.gz/doc/reST/api.rst -> x265_2.9.tar.gz/doc/reST/api.rst
Changed
51
1
2
* returns negative on error, 0 access unit were output.*/3
int x265_set_analysis_data(x265_encoder *encoder, x265_analysis_data *analysis_data, int poc, uint32_t cuBytes);4
5
+**x265_alloc_analysis_data()** may be used to allocate memory for the x265_analysis_data::6
+7
+ /* x265_alloc_analysis_data:8
+ * Allocate memory for the x265_analysis_data object's internal structures. */9
+ void x265_alloc_analysis_data(x265_param *param, x265_analysis_data* analysis);10
+11
+**x265_free_analysis_data()** may be used to free memory for the x265_analysis_data::12
+13
+ /* x265_free_analysis_data:14
+ * Free the allocated memory for x265_analysis_data object's internal structures. */15
+ void x265_free_analysis_data(x265_param *param, x265_analysis_data* analysis);16
+17
Pictures18
========19
20
21
* release library static allocations, reset configured CTU size */22
void x265_cleanup(void);23
24
+VMAF (Video Multi-Method Assessment Fusion)25
+==========================================26
+27
+If you set the ENABLE_LIBVMAF cmake option to ON, then x265 will report per frame28
+and aggregate VMAF score for the given input and dump the scores in csv file.29
+The user also need to specify the :option:`--recon` in command line to get the VMAF scores.30
+ 31
+ /* x265_calculate_vmafScore:32
+ * returns VMAF score for the input video.33
+ * This api must be called only after encoding was done. */34
+ double x265_calculate_vmafscore(x265_param*, x265_vmaf_data*);35
+36
+ /* x265_calculate_vmaf_framelevelscore:37
+ * returns VMAF score for each frame in a given input video. The frame level VMAF score does not include temporal scores. */38
+ double x265_calculate_vmaf_framelevelscore(x265_vmaf_framedata*);39
+ 40
+.. Note::41
42
+ When setting ENABLE_LIBVMAF cmake option to ON, it is recommended to43
+ also set ENABLE_SHARED to OFF to prevent build problems. 44
+ We only need the static library from these builds.45
+ 46
+ Binaries build with windows will not have VMAF support.47
+ 48
Multi-library Interface49
=======================50
51
x265_2.7.tar.gz/doc/reST/cli.rst -> x265_2.9.tar.gz/doc/reST/cli.rst
Changed
625
1
2
2. unable to open encoder3
3. unable to generate stream headers4
4. encoder abort5
- 6
+7
Logging/Statistic Options8
=========================9
10
11
**BufferFill** Bits available for the next frame. Includes bits carried12
over from the current frame.13
14
+ **BufferFillFinal** Buffer bits available after removing the frame out of CPB.15
+ 16
**Latency** Latency in terms of number of frames between when the frame 17
was given in and when the frame is given out.18
19
20
21
.. option:: --csv-log-level <integer>22
23
- Controls the level of detail (and size) of --csv log files24
- 25
- 0. summary **(default)**26
- 1. frame level logging27
- 2. frame level logging with performance statistics28
+ Controls the level of detail (and size) of --csv log files29
+30
+ 0. summary **(default)**31
+ 1. frame level logging32
+ 2. frame level logging with performance statistics33
34
.. option:: --ssim, --no-ssim35
36
37
"*" - same as default38
"none" - no thread pools are created, only frame parallelism possible39
"-" - same as "none"40
- "10" - allocate one pool, using up to 10 cores on node 041
+ "10" - allocate one pool, using up to 10 cores on all available nodes42
"-,+" - allocate one pool, using all cores on node 143
"+,-,+" - allocate one pool, using only cores on nodes 0 and 244
"+,-,+,-" - allocate one pool, using only cores on nodes 0 and 245
46
47
**CLI ONLY**48
49
+.. option:: --chunk-start <integer>50
+51
+ First frame of the chunk. Frames preceeding this in display order will52
+ be encoded, however, they will be discarded in the bitstream. This53
+ feature can be enabled only in closed GOP structures.54
+ Default 0 (disabled).55
+ 56
+.. option:: --chunk-end <integer>57
+58
+ Last frame of the chunk. Frames following this in display order will be59
+ used in taking lookahead decisions, but, they will not be encoded.60
+ This feature can be enabled only in closed GOP structures.61
+ Default 0 (disabled).62
+63
Profile, Level, Tier64
====================65
66
67
encoding options, the encoder will attempt to modify/set the right 68
encode specifications. If the encoder is unable to do so, this option69
will be turned OFF. Highly experimental.70
- 71
+72
Default: disabled73
- 74
+75
.. note::76
77
:option:`--profile`, :option:`--level-idc`, and78
79
Default 3.80
81
.. option:: --limit-modes, --no-limit-modes82
- 83
+84
When enabled, limit-modes will limit modes analyzed for each CU using cost 85
metrics from the 4 sub-CUs. When multiple inter modes like :option:`--rect`86
and/or :option:`--amp` are enabled, this feature will use motion cost 87
88
89
Default: enabled, disabled for :option:`--tune grain`90
91
+.. option:: --splitrd-skip, --no-splitrd-skip92
+93
+ Enable skipping split RD analysis when sum of split CU rdCost larger than one94
+ split CU rdCost for Intra CU. Default disabled.95
+96
.. option:: --fast-intra, --no-fast-intra97
98
Perform an initial scan of every fifth intra angular mode, then99
100
101
Note that --analysis-reuse-level must be paired with analysis-reuse-mode.102
103
- +--------------+------------------------------------------+104
- | Level | Description |105
- +==============+==========================================+106
- | 1 | Lookahead information |107
- +--------------+------------------------------------------+108
- | 2 to 4 | Level 1 + intra/inter modes, ref's |109
- +--------------+------------------------------------------+110
- | 5,6 and 9 | Level 2 + rect-amp |111
- +--------------+------------------------------------------+112
- | 7 | Level 5 + AVC size CU refinement |113
- +--------------+------------------------------------------+114
- | 8 | Level 5 + AVC size Full CU analysis-info |115
- +--------------+------------------------------------------+116
- | 10 | Level 5 + Full CU analysis-info |117
- +--------------+------------------------------------------+118
+ +--------------+------------------------------------------+119
+ | Level | Description |120
+ +==============+==========================================+121
+ | 1 | Lookahead information |122
+ +--------------+------------------------------------------+123
+ | 2 to 4 | Level 1 + intra/inter modes, ref's |124
+ +--------------+------------------------------------------+125
+ | 5 and 6 | Level 2 + rect-amp |126
+ +--------------+------------------------------------------+127
+ | 7 | Level 5 + AVC size CU refinement |128
+ +--------------+------------------------------------------+129
+ | 8 and 9 | Level 5 + AVC size Full CU analysis-info |130
+ +--------------+------------------------------------------+131
+ | 10 | Level 5 + Full CU analysis-info |132
+ +--------------+------------------------------------------+133
134
.. option:: --refine-mv-type <string>135
136
- Reuse MV information received through API call. Currently receives information for AVC size and the accepted 137
- string input is "avc". Default is disabled.138
+ Reuse MV information received through API call. Currently receives information for AVC size and the accepted 139
+ string input is "avc". Default is disabled.140
141
.. option:: --scale-factor142
143
- Factor by which input video is scaled down for analysis save mode.144
- This option should be coupled with analysis-reuse-mode option, --analysis-reuse-level 10.145
- The ctu size of load should be double the size of save. Default 0.146
+ Factor by which input video is scaled down for analysis save mode.147
+ This option should be coupled with analysis-reuse-mode option, 148
+ --analysis-reuse-level 10. The ctu size of load can either be the 149
+ same as that of save or double the size of save. Default 0.150
+151
+.. option:: --refine-intra <0..4>152
153
-.. option:: --refine-intra <0..3>154
- 155
Enables refinement of intra blocks in current encode. 156
157
Level 0 - Forces both mode and depth from the save encode.158
159
160
Level 3 - Perform analysis of intra modes for depth reused from first encode.161
162
- Default 0.163
+ Level 4 - Does not reuse any analysis information - redo analysis for the intra block.164
165
+ Default 0.166
+167
.. option:: --refine-inter <0..3>168
169
Enables refinement of inter blocks in current encode. 170
171
172
Default 0.173
174
+.. option:: --dynamic-refine, --no-dynamic-refine175
+176
+ Dynamically switches :option:`--refine-inter` levels 0-3 based on the content and 177
+ the encoder settings. It is recommended to use :option:`--refine-intra` 4 with dynamic 178
+ refinement. Default disabled.179
+180
.. option:: --refine-mv181
182
Enables refinement of motion vector for scaled video. Evaluates the best 183
motion vector by searching the surrounding eight integer and subpel pixel184
- positions.185
+ positions.186
187
Options which affect the transform unit quad-tree, sometimes referred to188
as the residual quad-tree (RQT).189
190
quad-tree begins at the same depth of the coded tree unit, but if the191
maximum TU size is smaller than the CU size then transform QT begins 192
at the depth of the max-tu-size. Default: 32.193
- 194
+195
.. option:: --dynamic-rd <0..4>196
- 197
+198
Increases the RD level at points where quality drops due to VBV rate 199
control enforcement. The number of CUs for which the RD is reconfigured 200
is determined based on the strength. Strength 1 gives the best FPS, 201
202
203
.. option:: --ssim-rd, --no-ssim-rd204
205
- Enable/Disable SSIM RDO. SSIM is a better perceptual quality assessment206
- method as compared to MSE. SSIM based RDO calculation is based on residual207
- divisive normalization scheme. This normalization is consistent with the 208
- luminance and contrast masking effect of Human Visual System. It is used209
- for mode selection during analysis of CTUs and can achieve significant 210
- gain in terms of objective quality metrics SSIM and PSNR. It only has effect211
- on presets which use RDO-based mode decisions (:option:`--rd` 3 and above).212
+ Enable/Disable SSIM RDO. SSIM is a better perceptual quality assessment213
+ method as compared to MSE. SSIM based RDO calculation is based on residual214
+ divisive normalization scheme. This normalization is consistent with the 215
+ luminance and contrast masking effect of Human Visual System. It is used216
+ for mode selection during analysis of CTUs and can achieve significant 217
+ gain in terms of objective quality metrics SSIM and PSNR. It only has effect218
+ on presets which use RDO-based mode decisions (:option:`--rd` 3 and above).219
220
Temporal / motion search options221
================================222
223
224
.. option:: --analyze-src-pics, --no-analyze-src-pics225
226
- Enalbe motion estimation with source frame pixels, in this mode, 227
- motion estimation can be computed independently. Default disabled.228
+ Enable motion estimation with source frame pixels, in this mode, 229
+ motion estimation can be computed independently. Default disabled.230
231
Spatial/intra options232
=====================233
234
235
.. option:: --ctu-info <0, 1, 2, 4, 6>236
237
- This value enables receiving CTU information asynchronously and determine reaction to the CTU information. Default 0.238
- 1: force the partitions if CTU information is present.239
- 2: functionality of (1) and reduce qp if CTU information has changed.240
- 4: functionality of (1) and force Inter modes when CTU Information has changed, merge/skip otherwise.241
- This option should be enabled only when planning to invoke the API function x265_encoder_ctu_info to copy ctu-info asynchronously. 242
- If enabled without calling the API function, the encoder will wait indefinitely.243
+ This value enables receiving CTU information asynchronously and determine reaction to the CTU information. Default 0.244
+ 1: force the partitions if CTU information is present.245
+ 2: functionality of (1) and reduce qp if CTU information has changed.246
+ 4: functionality of (1) and force Inter modes when CTU Information has changed, merge/skip otherwise.247
+ This option should be enabled only when planning to invoke the API function x265_encoder_ctu_info to copy ctu-info asynchronously. 248
+ If enabled without calling the API function, the encoder will wait indefinitely.249
250
.. option:: --intra-refresh251
252
253
Default 20254
255
**Range of values:** Between the maximum consecutive bframe count (:option:`--bframes`) and 250256
+257
.. option:: --gop-lookahead <integer>258
259
- Number of frames for GOP boundary decision lookahead. If a scenecut frame is found260
- within this from the gop boundary set by `--keyint`, the GOP will be extented until such a point,261
- otherwise the GOP will be terminated as set by `--keyint`. Default 0.262
+ Number of frames for GOP boundary decision lookahead. If a scenecut frame is found263
+ within this from the gop boundary set by `--keyint`, the GOP will be extented until such a point,264
+ otherwise the GOP will be terminated as set by `--keyint`. Default 0.265
266
- **Range of values:** Between 0 and (`--rc-lookahead` - mini-GOP length)267
+ **Range of values:** Between 0 and (`--rc-lookahead` - mini-GOP length)268
269
- It is recommended to have `--gop-lookahaed` less than `--min-keyint` as scenecuts beyond270
- `--min-keyint` are already being coded as keyframes.271
+ It is recommended to have `--gop-lookahaed` less than `--min-keyint` as scenecuts beyond272
+ `--min-keyint` are already being coded as keyframes.273
274
.. option:: --lookahead-slices <0..16>275
276
277
on systems with many threads.278
279
The encoder may internally lower the number of slices or disable280
- slicing to ensure each slice codes at least 10 16x16 rows of lowres281
- blocks to minimize the impact on quality. For example, for 720p and282
- 1080p videos, the number of slices is capped to 4 and 6, respectively.283
- For resolutions lesser than 720p, slicing is auto-disabled.284
- 285
- If slices are used in lookahead, they are logged in the list of tools286
- as *lslices*287
+ slicing to ensure each slice codes at least 10 16x16 rows of lowres288
+ blocks to minimize the impact on quality. For example, for 720p and289
+ 1080p videos, the number of slices is capped to 4 and 6, respectively.290
+ For resolutions lesser than 720p, slicing is auto-disabled.291
+292
+ If slices are used in lookahead, they are logged in the list of tools293
+ as *lslices*294
295
**Values:** 0 - disabled. 1 is the same as 0. Max 16.296
- Default: 8 for ultrafast, superfast, faster, fast, medium297
- 4 for slow, slower298
- disabled for veryslow, slower299
- 300
+ Default: 8 for ultrafast, superfast, faster, fast, medium301
+ 4 for slow, slower302
+ disabled for veryslow, slower303
+304
.. option:: --lookahead-threads <integer>305
306
- Use multiple worker threads dedicated to doing only lookahead instead of sharing307
- the worker threads with frame Encoders. A dedicated lookahead threadpool is created with the308
- specified number of worker threads. This can range from 0 upto half the309
- hardware threads available for encoding. Using too many threads for lookahead can starve310
- resources for frame Encoder and can harm performance. Default is 0 - disabled, Lookahead 311
+ Use multiple worker threads dedicated to doing only lookahead instead of sharing312
+ the worker threads with frame Encoders. A dedicated lookahead threadpool is created with the313
+ specified number of worker threads. This can range from 0 upto half the314
+ hardware threads available for encoding. Using too many threads for lookahead can starve315
+ resources for frame Encoder and can harm performance. Default is 0 - disabled, Lookahead 316
shares worker threads with other FrameEncoders . 317
318
**Values:** 0 - disabled(default). Max - Half of available hardware threads.319
- 320
+321
.. option:: --b-adapt <integer>322
323
Set the level of effort in determining B frame placement.324
325
.. option:: --b-pyramid, --no-b-pyramid326
327
Use B-frames as references, when possible. Default enabled328
- 329
+330
.. option:: --force-flush <integer>331
332
Force the encoder to flush frames. Default is 0.333
- 334
+335
Values:336
0 - flush the encoder only when all the input pictures are over.337
1 - flush all the frames even when the input is not over. 338
339
any given frame (ensuring a max QP). This is dangerous when CRF is340
used in combination with VBV as it may result in buffer underruns.341
Default disabled342
- 343
+344
.. option:: --crf-min <0..51.0>345
346
Specify an lower limit to the rate factor which may be assigned to347
348
Default 0.9349
350
**Range of values:** fractional: 0 - 1.0, or kbits: 2 .. bufsize351
- 352
+353
.. option:: --vbv-end <float>354
355
Final buffer emptiness. The portion of the decode buffer that must be 356
357
can specify the starting and ending state of the VBV buffer so that VBV 358
compliance can be maintained when chunks are independently encoded and 359
stitched together.360
- 361
+362
.. option:: --vbv-end-fr-adj <float>363
364
Frame from which qp has to be adjusted to achieve final decode buffer365
366
367
.. option:: --multi-pass-opt-analysis, --no-multi-pass-opt-analysis368
369
- Enable/Disable multipass analysis refinement along with multipass ratecontrol. Based on 370
- the information stored in pass 1, in subsequent passes analysis data is refined 371
- and also redundant steps are skipped.372
- In pass 1 analysis information like motion vector, depth, reference and prediction373
- modes of the final best CTU partition is stored for each CTU.374
- Multipass analysis refinement cannot be enabled when 'analysis-save/analysis-load' option375
- is enabled and both will be disabled when enabled together. This feature requires 'pmode/pme'376
- to be disabled and hence pmode/pme will be disabled when enabled at the same time.377
+ Enable/Disable multipass analysis refinement along with multipass ratecontrol. Based on 378
+ the information stored in pass 1, in subsequent passes analysis data is refined 379
+ and also redundant steps are skipped.380
+ In pass 1 analysis information like motion vector, depth, reference and prediction381
+ modes of the final best CTU partition is stored for each CTU.382
+ Multipass analysis refinement cannot be enabled when 'analysis-save/analysis-load' option383
+ is enabled and both will be disabled when enabled together. This feature requires 'pmode/pme'384
+ to be disabled and hence pmode/pme will be disabled when enabled at the same time.385
386
- Default: disabled.387
+ Default: disabled.388
389
.. option:: --multi-pass-opt-distortion, --no-multi-pass-opt-distortion390
391
- Enable/Disable multipass refinement of qp based on distortion data along with multipass392
- ratecontrol. In pass 1 distortion of best CTU partition is stored. CTUs with high393
- distortion get lower(negative)qp offsets and vice-versa for low distortion CTUs in pass 2.394
- This helps to improve the subjective quality.395
- Multipass refinement of qp cannot be enabled when 'analysis-save/analysis-load' option396
- is enabled and both will be disabled when enabled together. 'multi-pass-opt-distortion' 397
- requires 'pmode/pme' to be disabled and hence pmode/pme will be disabled when enabled along with it.398
+ Enable/Disable multipass refinement of qp based on distortion data along with multipass399
+ ratecontrol. In pass 1 distortion of best CTU partition is stored. CTUs with high400
+ distortion get lower(negative)qp offsets and vice-versa for low distortion CTUs in pass 2.401
+ This helps to improve the subjective quality.402
+ Multipass refinement of qp cannot be enabled when 'analysis-save/analysis-load' option403
+ is enabled and both will be disabled when enabled together. 'multi-pass-opt-distortion' 404
+ requires 'pmode/pme' to be disabled and hence pmode/pme will be disabled when enabled along with it.405
406
- Default: disabled.407
+ Default: disabled.408
409
.. option:: --strict-cbr, --no-strict-cbr410
- 411
+412
Enables stricter conditions to control bitrate deviance from the 413
target bitrate in ABR mode. Bit rate adherence is prioritised414
over quality. Rate tolerance is reduced to 50%. Default disabled.415
416
encoded frames to control QP. strict-cbr allows the encoder to be 417
more aggressive in hitting the target bitrate even for short segment 418
videos.419
- 420
+421
.. option:: --cbqpoffs <integer>422
423
Offset of Cb chroma QP from the luma QP selected by rate control.424
425
426
qComp sets the quantizer curve compression factor. It weights the427
frame quantizer based on the complexity of residual (measured by428
- lookahead). Default value is 0.6. Increasing it to 1 will429
- effectively generate CQP430
+ lookahead). It's value must be between 0.5 and 1.0. Default value is431
+ 0.6. Increasing it to 1.0 will effectively generate CQP.432
433
.. option:: --qpstep <integer>434
435
- The maximum single adjustment in QP allowed to rate control. Default436
- 4437
- 438
+ The maximum single adjustment in QP allowed to rate control. Default 4439
+440
.. option:: --qpmin <integer>441
442
sets a hard lower limit on QP allowed to ratecontrol. Default 0443
444
.. option:: --qpmax <integer>445
446
sets a hard upper limit on QP allowed to ratecontrol. Default 69447
- 448
+449
.. option:: --rc-grain, --no-rc-grain450
451
- Enables a specialised ratecontrol algorithm for film grain content. This 452
- parameter strictly minimises QP fluctuations within and across frames 453
- and removes pulsing of grain. Default disabled. 454
- Enabled when :option:'--tune' grain is applied. It is highly recommended 455
- that this option is used through the tune grain feature where a combination 456
- of param options are used to improve visual quality.457
- 458
+ Enables a specialised ratecontrol algorithm for film grain content. This 459
+ parameter strictly minimises QP fluctuations within and across frames 460
+ and removes pulsing of grain. Default disabled. 461
+ Enabled when :option:'--tune' grain is applied. It is highly recommended 462
+ that this option is used through the tune grain feature where a combination 463
+ of param options are used to improve visual quality.464
+465
.. option:: --const-vbv, --no-const-vbv466
467
- Enables VBV algorithm to be consistent across runs. Default disabled. 468
- Enabled when :option:'--tune' grain is applied.469
- 470
+ Enables VBV algorithm to be consistent across runs. Default disabled. 471
+ Enabled when :option:'--tune' grain is applied.472
+473
.. option:: --qblur <float>474
475
Temporally blur quants. Default 0.5476
477
HEVC specifies a default set of scaling lists which may be enabled478
without requiring them to be signaled in the SPS. Those scaling479
lists can be enabled via :option:`--scaling-list` *default*.480
- 481
+482
All other strings indicate a filename containing custom scaling483
lists in the HM format. The encode will abort if the file is not484
- parsed correctly. Custom lists must be signaled in the SPS485
+ parsed correctly. Custom lists must be signaled in the SPS. A sample486
+ scaling list file is available in `the downloads page <https://bitbucket.org/multicoreware/x265/downloads/reference_scalinglist.txt>`_487
488
.. option:: --lambda-file <filename>489
490
Specify a text file containing values for x265_lambda_tab and491
x265_lambda2_tab. Each table requires MAX_MAX_QP+1 (70) float492
values.493
- 494
+495
The text file syntax is simple. Comma is considered to be496
white-space. All white-space is ignored. Lines must be less than 2k497
bytes in length. Content following hash (#) characters are ignored.498
499
vectors and splits) and less on residual. This feature is intended500
for experimentation.501
502
+.. option:: --max-ausize-factor <float>503
+504
+ It controls the maximum AU size defined in specification. It represents505
+ the percentage of maximum AU size used. Default is 1. Range is 0.5 to 1.506
+507
Loop filters508
============509
510
511
7. smpte240m512
8. film513
9. bt2020514
- 10. smpte428515
- 11. smpte431516
- 12. smpte432517
+ 10. smpte428518
+ 11. smpte431519
+ 12. smpte432520
521
.. option:: --transfer <integer|string>522
523
524
8. YCgCo525
9. bt2020nc526
10. bt2020c527
- 11. smpte2085528
- 12. chroma-derived-nc529
- 13. chroma-derived-c530
- 14. ictcp531
+ 11. smpte2085532
+ 12. chroma-derived-nc533
+ 13. chroma-derived-c534
+ 14. ictcp535
536
.. option:: --chromaloc <0..5>537
538
539
automatically when :option:`--master-display` or :option:`--max-cll` is540
specified. Useful when there is a desire to signal 0 values for max-cll541
and max-fall. Default disabled.542
- 543
+544
.. option:: --hdr-opt, --no-hdr-opt545
546
Add luma and chroma offsets for HDR/WCG content.547
Input video should be 10 bit 4:2:0. Applicable for HDR content. It is recommended548
that AQ-mode be enabled along with this feature. Default disabled.549
- 550
+551
.. option:: --dhdr10-info <filename>552
553
Inserts tone mapping information as an SEI message. It takes as input, 554
555
Maximum luma value allowed for input pictures. Any values above max-luma556
are clipped. No default.557
558
+.. option:: --nalu-file <filename>559
+560
+ Text file containing userSEI in POC order : <POC><space><PREFIX><space><NAL UNIT TYPE>/<SEI TYPE><space><SEI Payload>561
+ Parse the input file specified and inserts SEI messages into the bitstream. 562
+ Currently, we support only PREFIX SEI messages. This is an "application-only" feature.563
+564
+.. option:: --atc-sei <integer>565
+566
+ Emit the alternative transfer characteristics SEI message where the integer567
+ is the preferred transfer characteristics. Required for HLG (Hybrid Log Gamma)568
+ signalling. Not signalled by default.569
+570
+.. option:: --pic-struct <integer>571
+572
+ Set the picture structure and emits it in the picture timing SEI message.573
+ Values in the range 0..12. See D.3.3 of the HEVC spec. for a detailed explanation.574
+ Required for HLG (Hybrid Log Gamma) signalling. Not signalled by default.575
+576
Bitstream options577
=================578
579
580
581
.. option:: --log2-max-poc-lsb <integer>582
583
- Maximum of the picture order count. Default 8584
+ Maximum of the picture order count. Default 8585
586
.. option:: --vui-timing-info, --no-vui-timing-info587
588
589
590
Only effective at RD levels 5 and 6591
592
+.. option:: --idr-recovery-sei, --no-idr-recovery-sei593
+ Emit RecoveryPoint info as sei in bitstream for each IDR frame. Default disabled.594
+595
+.. option:: --single-sei, --no-single-sei596
+ Emit SEI messages in a single NAL unit instead of multiple NALs. Default disabled.597
+ When HRD SEI is enabled the HM decoder will throw a warning.598
+599
DCT Approximations600
=================601
602
.. option:: --lowpass-dct603
604
- If enabled, x265 will use low-pass subband dct approximation instead of the605
- standard dct for 16x16 and 32x32 blocks. This approximation is less computational 606
- intensive but it generates truncated coefficient matrixes for the transformed block. 607
- Empirical analysis shows marginal loss in compression and performance gains up to 10%,608
- paticularly at moderate bit-rates.609
+ If enabled, x265 will use low-pass subband dct approximation instead of the610
+ standard dct for 16x16 and 32x32 blocks. This approximation is less computational 611
+ intensive but it generates truncated coefficient matrixes for the transformed block. 612
+ Empirical analysis shows marginal loss in compression and performance gains up to 10%,613
+ paticularly at moderate bit-rates.614
615
- This approximation should be considered for platforms with performance and time 616
- constrains.617
+ This approximation should be considered for platforms with performance and time 618
+ constrains.619
620
- Default disabled. **Experimental feature**621
+ Default disabled. **Experimental feature**622
623
Debugging options624
=================625
x265_2.7.tar.gz/doc/reST/presets.rst -> x265_2.9.tar.gz/doc/reST/presets.rst
Changed
13
1
2
that strictly minimises QP fluctuations across frames, while still allowing 3
the encoder to hit bitrate targets and VBV buffer limits (with a slightly 4
higher margin of error than normal). It is highly recommended that this 5
-algorithm is used only through the :option:`--tune` *grain* feature.6
+algorithm is used only through the :option:`--tune` *grain* feature. 7
+Overriding the `--tune` *grain* settings might result in grain strobing, especially8
+when enabling features like :option:`--aq-mode` and :option:`--cutree` that modify9
+per-block QPs within a given frame.10
11
Fast Decode12
~~~~~~~~~~~13
x265_2.7.tar.gz/doc/reST/releasenotes.rst -> x265_2.9.tar.gz/doc/reST/releasenotes.rst
Changed
71
1
2
Release Notes3
*************4
5
+Version 2.96
+===========7
+8
+Release date - 05/10/20189
+10
+New features11
+-------------12
+1. Support for chunked encoding13
+14
+ :option:`--chunk-start and --chunk-end` 15
+ Frames preceding first frame of chunk in display order will be encoded, however, they will be discarded in the bitstream.16
+ Frames following last frame of the chunk in display order will be used in taking lookahead decisions, but, they will not be encoded. 17
+ This feature can be enabled only in closed GOP structures. Default disabled.18
+19
+2. Support for HDR10+ version 1 SEI messages.20
+21
+Encoder enhancements22
+--------------------23
+1. Create API function for allocating and freeing x265_analysis_data.24
+2. CEA 608/708 support: Read SEI messages from text file and encode it using userSEI message.25
+26
+Bug fixes27
+---------28
+1. Disable noise reduction when vbv is enabled.29
+2. Support minLuma and maxLuma values changed by the commandline.30
+31
+Version 2.832
+===========33
+34
+Release date - 21/05/201835
+36
+New features37
+-------------38
+1. :option:`--asm avx512` used to enable AVX-512 in x265. Default disabled. 39
+ For 4K main10 high-quality encoding, we are seeing good gains; for other resolutions and presets, we don't recommend using this setting for now.40
+41
+2. :option:`--dynamic-refine` dynamically switches between different inter refine levels. Default disabled.42
+ It is recommended to use :option:`--refine-intra 4' with dynamic refinement for a better trade-off between encode efficiency and performance than using static refinement.43
+44
+3. :option:`--single-sei`45
+ Encode SEI messages in a single NAL unit instead of multiple NAL units. Default disabled. 46
+47
+4. :option:`--max-ausize-factor` controls the maximum AU size defined in HEVC specification.48
+ It represents the percentage of maximum AU size used. Default is 1. 49
+ 50
+5. VMAF (Video Multi-Method Assessment Fusion)51
+ Added VMAF support for objective quality measurement of a video sequence. 52
+ Enable cmake option ENABLE_LIBVMAF to report per frame and aggregate VMAF score. The frame level VMAF score does not include temporal scores.53
+ This is supported only on linux for now.54
+ 55
+Encoder enhancements56
+--------------------57
+1. Introduced refine-intra level 4 to improve quality. 58
+2. Support for HLG-graded content and pic_struct in SEI message.59
+60
+Bug Fixes61
+---------62
+1. Fix 32 bit build error (using CMAKE GUI) in Linux.63
+2. Fix 32 bit build error for asm primitives.64
+3. Fix build error on mac OS.65
+4. Fix VBV Lookahead in analysis load to achieve target bitrate.66
+67
+68
Version 2.769
===========70
71
x265_2.7.tar.gz/source/CMakeLists.txt -> x265_2.9.tar.gz/source/CMakeLists.txt
Changed
57
1
2
option(STATIC_LINK_CRT "Statically link C runtime for release builds" OFF)3
mark_as_advanced(FPROFILE_USE FPROFILE_GENERATE NATIVE_BUILD)4
# X265_BUILD must be incremented each time the public API is changed5
-set(X265_BUILD 151)6
+set(X265_BUILD 165)7
configure_file("${PROJECT_SOURCE_DIR}/x265.def.in"8
"${PROJECT_BINARY_DIR}/x265.def")9
configure_file("${PROJECT_SOURCE_DIR}/x265_config.h.in"10
11
if("${SYSPROC}" STREQUAL "" OR X86MATCH GREATER "-1")12
set(X86 1)13
add_definitions(-DX265_ARCH_X86=1)14
- if("${CMAKE_SIZEOF_VOID_P}" MATCHES 8)15
+ if(CMAKE_CXX_FLAGS STREQUAL "-m32")16
+ message(STATUS "Detected x86 target processor")17
+ elseif("${CMAKE_SIZEOF_VOID_P}" MATCHES 8)18
set(X64 1)19
add_definitions(-DX86_64=1)20
message(STATUS "Detected x86_64 target processor")21
- else()22
- message(STATUS "Detected x86 target processor")23
endif()24
elseif(POWERMATCH GREATER "-1")25
message(STATUS "Detected POWER target processor")26
27
if(NO_ATOMICS)28
add_definitions(-DNO_ATOMICS=1)29
endif(NO_ATOMICS)30
+ find_library(VMAF vmaf)31
+ option(ENABLE_LIBVMAF "Enable VMAF" OFF)32
+ if(ENABLE_LIBVMAF)33
+ add_definitions(-DENABLE_LIBVMAF)34
+ endif()35
endif(UNIX)36
37
if(X64 AND NOT WIN32)38
39
if(EXTRA_LIB)40
target_link_libraries(x265-static ${EXTRA_LIB})41
endif()42
+if(ENABLE_LIBVMAF)43
+ target_link_libraries(x265-static ${VMAF})44
+endif()45
install(TARGETS x265-static46
LIBRARY DESTINATION ${LIB_INSTALL_DIR}47
ARCHIVE DESTINATION ${LIB_INSTALL_DIR})48
49
ARCHIVE DESTINATION ${LIB_INSTALL_DIR})50
endif()51
install(FILES x265.h "${PROJECT_BINARY_DIR}/x265_config.h" DESTINATION include)52
-if(WIN32)53
+if((WIN32 AND ENABLE_CLI) OR (WIN32 AND ENABLE_SHARED))54
if(MSVC_IDE)55
install(FILES "${PROJECT_BINARY_DIR}/Debug/x265.pdb" DESTINATION ${BIN_INSTALL_DIR} CONFIGURATIONS Debug)56
install(FILES "${PROJECT_BINARY_DIR}/RelWithDebInfo/x265.pdb" DESTINATION ${BIN_INSTALL_DIR} CONFIGURATIONS RelWithDebInfo)57
x265_2.7.tar.gz/source/common/common.cpp -> x265_2.9.tar.gz/source/common/common.cpp
Changed
10
1
2
#endif3
}4
5
-#define X265_ALIGNBYTES 326
+#define X265_ALIGNBYTES 647
8
#if _WIN329
#if defined(__MINGW32__) && !defined(__MINGW64_VERSION_MAJOR)10
x265_2.7.tar.gz/source/common/common.h -> x265_2.9.tar.gz/source/common/common.h
Changed
26
1
2
#define ALIGN_VAR_8(T, var) T var __attribute__((aligned(8)))3
#define ALIGN_VAR_16(T, var) T var __attribute__((aligned(16)))4
#define ALIGN_VAR_32(T, var) T var __attribute__((aligned(32)))5
+#define ALIGN_VAR_64(T, var) T var __attribute__((aligned(64)))6
#if defined(__MINGW32__)7
#define fseeko fseeko648
#define ftello ftello649
10
#define ALIGN_VAR_8(T, var) __declspec(align(8)) T var11
#define ALIGN_VAR_16(T, var) __declspec(align(16)) T var12
#define ALIGN_VAR_32(T, var) __declspec(align(32)) T var13
+#define ALIGN_VAR_64(T, var) __declspec(align(64)) T var14
#define fseeko _fseeki6415
#define ftello _ftelli6416
#endif // if defined(__GNUC__)17
18
#define START_CODE_OVERHEAD 3 19
#define FILLER_OVERHEAD (NAL_TYPE_OVERHEAD + START_CODE_OVERHEAD + 1)20
21
+#define MAX_NUM_DYN_REFINE (NUM_CU_DEPTH * X265_REFINE_INTER_LEVELS)22
+23
namespace X265_NS {24
25
enum { SAO_NUM_OFFSET = 4 };26
x265_2.7.tar.gz/source/common/cpu.cpp -> x265_2.9.tar.gz/source/common/cpu.cpp
Changed
200
1
2
#endif // if X265_ARCH_ARM3
4
namespace X265_NS {5
+static bool enable512 = false;6
const cpu_name_t cpu_names[] =7
{8
#if X265_ARCH_X869
-#define MMX2 X265_CPU_MMX | X265_CPU_MMX2 | X265_CPU_CMOV10
+#define MMX2 X265_CPU_MMX | X265_CPU_MMX211
{ "MMX2", MMX2 },12
{ "MMXEXT", MMX2 },13
{ "SSE", MMX2 | X265_CPU_SSE },14
15
{ "BMI2", AVX | X265_CPU_LZCNT | X265_CPU_BMI1 | X265_CPU_BMI2 },16
#define AVX2 AVX | X265_CPU_FMA3 | X265_CPU_LZCNT | X265_CPU_BMI1 | X265_CPU_BMI2 | X265_CPU_AVX217
{ "AVX2", AVX2},18
+ { "AVX512", AVX2 | X265_CPU_AVX512 },19
#undef AVX220
#undef AVX21
#undef SSE222
#undef MMX223
{ "Cache32", X265_CPU_CACHELINE_32 },24
{ "Cache64", X265_CPU_CACHELINE_64 },25
- { "SlowCTZ", X265_CPU_SLOW_CTZ },26
{ "SlowAtom", X265_CPU_SLOW_ATOM },27
{ "SlowPshufb", X265_CPU_SLOW_PSHUFB },28
{ "SlowPalignr", X265_CPU_SLOW_PALIGNR },29
30
/* cpu-a.asm */31
int PFX(cpu_cpuid_test)(void);32
void PFX(cpu_cpuid)(uint32_t op, uint32_t *eax, uint32_t *ebx, uint32_t *ecx, uint32_t *edx);33
-void PFX(cpu_xgetbv)(uint32_t op, uint32_t *eax, uint32_t *edx);34
+uint64_t PFX(cpu_xgetbv)(int xcr);35
}36
37
#if defined(_MSC_VER)38
#pragma warning(disable: 4309) // truncation of constant value39
#endif40
41
-uint32_t cpu_detect(void)42
+bool detect512()43
+{44
+ return(enable512);45
+}46
+uint32_t cpu_detect(bool benableavx512 )47
{48
- uint32_t cpu = 0;49
50
+ uint32_t cpu = 0; 51
uint32_t eax, ebx, ecx, edx;52
uint32_t vendor[4] = { 0 };53
uint32_t max_extended_cap, max_basic_cap;54
+ uint64_t xcr0 = 0;55
56
#if !X86_6457
if (!PFX(cpu_cpuid_test)())58
return 0;59
#endif60
61
- PFX(cpu_cpuid)(0, &eax, vendor + 0, vendor + 2, vendor + 1);62
- max_basic_cap = eax;63
+ PFX(cpu_cpuid)(0, &max_basic_cap, vendor + 0, vendor + 2, vendor + 1);64
if (max_basic_cap == 0)65
return 0;66
67
68
return cpu;69
if (edx & 0x02000000)70
cpu |= X265_CPU_MMX2 | X265_CPU_SSE;71
- if (edx & 0x00008000)72
- cpu |= X265_CPU_CMOV;73
- else74
- return cpu;75
if (edx & 0x04000000)76
cpu |= X265_CPU_SSE2;77
if (ecx & 0x00000001)78
cpu |= X265_CPU_SSE3;79
if (ecx & 0x00000200)80
- cpu |= X265_CPU_SSSE3;81
+ cpu |= X265_CPU_SSSE3 | X265_CPU_SSE2_IS_FAST;82
if (ecx & 0x00080000)83
cpu |= X265_CPU_SSE4;84
if (ecx & 0x00100000)85
cpu |= X265_CPU_SSE42;86
- /* Check OXSAVE and AVX bits */87
- if ((ecx & 0x18000000) == 0x18000000)88
+89
+ if (ecx & 0x08000000) /* XGETBV supported and XSAVE enabled by OS */90
{91
/* Check for OS support */92
- PFX(cpu_xgetbv)(0, &eax, &edx);93
- if ((eax & 0x6) == 0x6)94
+ xcr0 = PFX(cpu_xgetbv)(0);95
+ if ((xcr0 & 0x6) == 0x6) /* XMM/YMM state */96
{97
+ if (ecx & 0x10000000)98
cpu |= X265_CPU_AVX;99
if (ecx & 0x00001000)100
cpu |= X265_CPU_FMA3;101
102
{103
PFX(cpu_cpuid)(7, &eax, &ebx, &ecx, &edx);104
/* AVX2 requires OS support, but BMI1/2 don't. */105
- if ((cpu & X265_CPU_AVX) && (ebx & 0x00000020))106
- cpu |= X265_CPU_AVX2;107
if (ebx & 0x00000008)108
- {109
cpu |= X265_CPU_BMI1;110
- if (ebx & 0x00000100)111
- cpu |= X265_CPU_BMI2;112
+ if (ebx & 0x00000100)113
+ cpu |= X265_CPU_BMI2;114
+115
+ if ((xcr0 & 0x6) == 0x6) /* XMM/YMM state */116
+ {117
+ if (ebx & 0x00000020)118
+ cpu |= X265_CPU_AVX2;119
+ if (benableavx512)120
+ {121
+ if ((xcr0 & 0xE0) == 0xE0) /* OPMASK/ZMM state */122
+ {123
+ if ((ebx & 0xD0030000) == 0xD0030000)124
+ {125
+ cpu |= X265_CPU_AVX512;126
+ enable512 = true;127
+ }128
+ }129
+ }130
}131
}132
133
- if (cpu & X265_CPU_SSSE3)134
- cpu |= X265_CPU_SSE2_IS_FAST;135
-136
PFX(cpu_cpuid)(0x80000000, &eax, &ebx, &ecx, &edx);137
max_extended_cap = eax;138
139
140
{141
if (edx & 0x00400000)142
cpu |= X265_CPU_MMX2;143
- if (!(cpu & X265_CPU_LZCNT))144
- cpu |= X265_CPU_SLOW_CTZ;145
if ((cpu & X265_CPU_SSE2) && !(cpu & X265_CPU_SSE2_IS_FAST))146
cpu |= X265_CPU_SSE2_IS_SLOW; /* AMD CPUs come in two types: terrible at SSE and great at it */147
}148
149
int model = ((eax >> 4) & 0xf) + ((eax >> 12) & 0xf0);150
if (family == 6)151
{152
- /* 6/9 (pentium-m "banias"), 6/13 (pentium-m "dothan"), and 6/14 (core1 "yonah")153
- * theoretically support sse2, but it's significantly slower than mmx for154
- * almost all of x264's functions, so let's just pretend they don't. */155
- if (model == 9 || model == 13 || model == 14)156
- {157
- cpu &= ~(X265_CPU_SSE2 | X265_CPU_SSE3);158
- X265_CHECK(!(cpu & (X265_CPU_SSSE3 | X265_CPU_SSE4)), "unexpected CPU ID %d\n", cpu);159
- }160
/* Detect Atom CPU */161
- else if (model == 28)162
+ if (model == 28)163
{164
cpu |= X265_CPU_SLOW_ATOM;165
- cpu |= X265_CPU_SLOW_CTZ;166
cpu |= X265_CPU_SLOW_PSHUFB;167
}168
169
170
int PFX(cpu_fast_neon_mrc_test)(void);171
}172
173
-uint32_t cpu_detect(void)174
+uint32_t cpu_detect(bool benableavx512)175
{176
int flags = 0;177
178
179
180
#elif X265_ARCH_POWER8181
182
-uint32_t cpu_detect(void)183
+uint32_t cpu_detect(bool benableavx512)184
{185
#if HAVE_ALTIVEC186
return X265_CPU_ALTIVEC;187
188
189
#else // if X265_ARCH_POWER8190
191
-uint32_t cpu_detect(void)192
+uint32_t cpu_detect(bool benableavx512)193
{194
return 0;195
}196
197
#endif // if X265_ARCH_X86198
}199
+200
x265_2.7.tar.gz/source/common/cpu.h -> x265_2.9.tar.gz/source/common/cpu.h
Changed
19
1
2
#define X265_CPU_H3
4
#include "common.h"5
-6
/* All assembly functions are prefixed with X265_NS (macro expanded) */7
#define PFX3(prefix, name) prefix ## _ ## name8
#define PFX2(prefix, name) PFX3(prefix, name)9
10
#endif11
12
namespace X265_NS {13
-uint32_t cpu_detect(void);14
+uint32_t cpu_detect(bool);15
+bool detect512();16
17
struct cpu_name_t18
{19
x265_2.7.tar.gz/source/common/cudata.cpp -> x265_2.9.tar.gz/source/common/cudata.cpp
Changed
29
1
2
dir |= (1 << list);3
candMvField[count][list].mv = colmv;4
candMvField[count][list].refIdx = refIdx;5
- if (m_encData->m_param->scaleFactor && m_encData->m_param->analysisSave && m_log2CUSize[0] < 4)6
- {7
- MV dist(MAX_MV, MAX_MV);8
- candMvField[count][list].mv = dist;9
- }10
}11
}12
13
14
15
int curRefPOC = m_slice->m_refPOCList[picList][refIdx];16
int curPOC = m_slice->m_poc;17
-18
- if (m_encData->m_param->scaleFactor && m_encData->m_param->analysisSave && (m_log2CUSize[0] < 4))19
- {20
- MV dist(MAX_MV, MAX_MV);21
- pmv[numMvc++] = amvpCand[num++] = dist;22
- }23
- else24
- pmv[numMvc++] = amvpCand[num++] = scaleMvByPOCDist(neighbours[MD_COLLOCATED].mv[picList], curPOC, curRefPOC, colPOC, colRefPOC);25
+ pmv[numMvc++] = amvpCand[num++] = scaleMvByPOCDist(neighbours[MD_COLLOCATED].mv[picList], curPOC, curRefPOC, colPOC, colRefPOC);26
}27
}28
29
x265_2.7.tar.gz/source/common/cudata.h -> x265_2.9.tar.gz/source/common/cudata.h
Changed
27
1
2
uint64_t m_fAc_den[3];3
uint64_t m_fDc_den[3];4
5
+ /* Feature values per CTU for dynamic refinement */6
+ uint64_t* m_collectCURd;7
+ uint32_t* m_collectCUVariance;8
+ uint32_t* m_collectCUCount;9
+10
CUData();11
12
void initialize(const CUDataMemPool& dataPool, uint32_t depth, const x265_param& param, int instance);13
14
coeff_t* trCoeffMemBlock;15
MV* mvMemBlock;16
sse_t* distortionMemBlock;17
+ uint64_t* dynRefineRdBlock;18
+ uint32_t* dynRefCntBlock;19
+ uint32_t* dynRefVarBlock;20
21
- CUDataMemPool() { charMemBlock = NULL; trCoeffMemBlock = NULL; mvMemBlock = NULL; distortionMemBlock = NULL; }22
+ CUDataMemPool() { charMemBlock = NULL; trCoeffMemBlock = NULL; mvMemBlock = NULL; distortionMemBlock = NULL; 23
+ dynRefineRdBlock = NULL; dynRefCntBlock = NULL; dynRefVarBlock = NULL;}24
25
bool create(uint32_t depth, uint32_t csp, uint32_t numInstances, const x265_param& param)26
{27
x265_2.7.tar.gz/source/common/dct.cpp -> x265_2.9.tar.gz/source/common/dct.cpp
Changed
130
1
2
sum += sbacGetEntropyBits(mstate, firstC2Flag);3
}4
}5
-6
return (sum & 0x00FFFFFF) + (c1 << 26) + (firstC2Idx << 28);7
}8
+template<int log2TrSize>9
+static void nonPsyRdoQuant_c(int16_t *m_resiDctCoeff, int64_t *costUncoded, int64_t *totalUncodedCost, int64_t *totalRdCost, uint32_t blkPos)10
+{11
+ const int transformShift = MAX_TR_DYNAMIC_RANGE - X265_DEPTH - log2TrSize; /* Represents scaling through forward transform */12
+ const int scaleBits = SCALE_BITS - 2 * transformShift;13
+ const uint32_t trSize = 1 << log2TrSize;14
+15
+ for (int y = 0; y < MLS_CG_SIZE; y++)16
+ {17
+ for (int x = 0; x < MLS_CG_SIZE; x++)18
+ {19
+ int64_t signCoef = m_resiDctCoeff[blkPos + x]; /* pre-quantization DCT coeff */20
+ costUncoded[blkPos + x] = static_cast<int64_t>((double)((signCoef * signCoef) << scaleBits));21
+ *totalUncodedCost += costUncoded[blkPos + x];22
+ *totalRdCost += costUncoded[blkPos + x];23
+ }24
+ blkPos += trSize;25
+ }26
+}27
+template<int log2TrSize>28
+static void psyRdoQuant_c(int16_t *m_resiDctCoeff, int16_t *m_fencDctCoeff, int64_t *costUncoded, int64_t *totalUncodedCost, int64_t *totalRdCost, int64_t *psyScale, uint32_t blkPos)29
+{30
+ const int transformShift = MAX_TR_DYNAMIC_RANGE - X265_DEPTH - log2TrSize; /* Represents scaling through forward transform */31
+ const int scaleBits = SCALE_BITS - 2 * transformShift;32
+ const uint32_t trSize = 1 << log2TrSize;33
+ int max = X265_MAX(0, (2 * transformShift + 1));34
+35
+ for (int y = 0; y < MLS_CG_SIZE; y++)36
+ {37
+ for (int x = 0; x < MLS_CG_SIZE; x++)38
+ {39
+ int64_t signCoef = m_resiDctCoeff[blkPos + x]; /* pre-quantization DCT coeff */40
+ int64_t predictedCoef = m_fencDctCoeff[blkPos + x] - signCoef; /* predicted DCT = source DCT - residual DCT*/41
+42
+ costUncoded[blkPos + x] = static_cast<int64_t>((double)((signCoef * signCoef) << scaleBits));43
+44
+ /* when no residual coefficient is coded, predicted coef == recon coef */45
+ costUncoded[blkPos + x] -= static_cast<int64_t>((double)(((*psyScale) * predictedCoef) >> max));46
+47
+ *totalUncodedCost += costUncoded[blkPos + x];48
+ *totalRdCost += costUncoded[blkPos + x];49
+ }50
+ blkPos += trSize;51
+ }52
+}53
+template<int log2TrSize>54
+static void psyRdoQuant_c_1(int16_t *m_resiDctCoeff, /*int16_t *m_fencDctCoeff, */ int64_t *costUncoded, int64_t *totalUncodedCost, int64_t *totalRdCost, /* int64_t *psyScale,*/ uint32_t blkPos)55
+{56
+ const int transformShift = MAX_TR_DYNAMIC_RANGE - X265_DEPTH - log2TrSize; /* Represents scaling through forward transform */57
+ const int scaleBits = SCALE_BITS - 2 * transformShift;58
+ const uint32_t trSize = 1 << log2TrSize;59
+60
+ for (int y = 0; y < MLS_CG_SIZE; y++)61
+ {62
+ for (int x = 0; x < MLS_CG_SIZE; x++)63
+ {64
+ int64_t signCoef = m_resiDctCoeff[blkPos + x]; /* pre-quantization DCT coeff */65
+ costUncoded[blkPos + x] = static_cast<int64_t>((double)((signCoef * signCoef) << scaleBits));66
+ *totalUncodedCost += costUncoded[blkPos + x];67
+ *totalRdCost += costUncoded[blkPos + x];68
+ }69
+ blkPos += trSize;70
+ }71
+}72
+template<int log2TrSize>73
+static void psyRdoQuant_c_2(int16_t *m_resiDctCoeff, int16_t *m_fencDctCoeff, int64_t *costUncoded, int64_t *totalUncodedCost, int64_t *totalRdCost, int64_t *psyScale, uint32_t blkPos)74
+{75
+ const int transformShift = MAX_TR_DYNAMIC_RANGE - X265_DEPTH - log2TrSize; /* Represents scaling through forward transform */76
+77
+ const uint32_t trSize = 1 << log2TrSize;78
+ int max = X265_MAX(0, (2 * transformShift + 1));79
+80
+ for (int y = 0; y < MLS_CG_SIZE; y++)81
+ {82
+ for (int x = 0; x < MLS_CG_SIZE; x++)83
+ {84
+ int64_t signCoef = m_resiDctCoeff[blkPos + x]; /* pre-quantization DCT coeff */85
+ int64_t predictedCoef = m_fencDctCoeff[blkPos + x] - signCoef; /* predicted DCT = source DCT - residual DCT*/86
+ costUncoded[blkPos + x] -= static_cast<int64_t>((double)(((*psyScale) * predictedCoef) >> max));87
+ *totalUncodedCost += costUncoded[blkPos + x];88
+ *totalRdCost += costUncoded[blkPos + x];89
+ }90
+ blkPos += trSize;91
+ }92
+}93
94
namespace X265_NS {95
// x265 private namespace96
-97
void setupDCTPrimitives_c(EncoderPrimitives& p)98
{99
p.dequant_scaling = dequant_scaling_c;100
p.dequant_normal = dequant_normal_c;101
p.quant = quant_c;102
p.nquant = nquant_c;103
+ p.cu[BLOCK_4x4].nonPsyRdoQuant = nonPsyRdoQuant_c<2>;104
+ p.cu[BLOCK_8x8].nonPsyRdoQuant = nonPsyRdoQuant_c<3>;105
+ p.cu[BLOCK_16x16].nonPsyRdoQuant = nonPsyRdoQuant_c<4>;106
+ p.cu[BLOCK_32x32].nonPsyRdoQuant = nonPsyRdoQuant_c<5>;107
+ p.cu[BLOCK_4x4].psyRdoQuant = psyRdoQuant_c<2>;108
+ p.cu[BLOCK_8x8].psyRdoQuant = psyRdoQuant_c<3>;109
+ p.cu[BLOCK_16x16].psyRdoQuant = psyRdoQuant_c<4>;110
+ p.cu[BLOCK_32x32].psyRdoQuant = psyRdoQuant_c<5>;111
p.dst4x4 = dst4_c;112
p.cu[BLOCK_4x4].dct = dct4_c;113
p.cu[BLOCK_8x8].dct = dct8_c;114
115
p.cu[BLOCK_8x8].copy_cnt = copy_count<8>;116
p.cu[BLOCK_16x16].copy_cnt = copy_count<16>;117
p.cu[BLOCK_32x32].copy_cnt = copy_count<32>;118
-119
+ p.cu[BLOCK_4x4].psyRdoQuant_1p = psyRdoQuant_c_1<2>;120
+ p.cu[BLOCK_4x4].psyRdoQuant_2p = psyRdoQuant_c_2<2>;121
+ p.cu[BLOCK_8x8].psyRdoQuant_1p = psyRdoQuant_c_1<3>;122
+ p.cu[BLOCK_8x8].psyRdoQuant_2p = psyRdoQuant_c_2<3>;123
+ p.cu[BLOCK_16x16].psyRdoQuant_1p = psyRdoQuant_c_1<4>;124
+ p.cu[BLOCK_16x16].psyRdoQuant_2p = psyRdoQuant_c_2<4>;125
+ p.cu[BLOCK_32x32].psyRdoQuant_1p = psyRdoQuant_c_1<5>;126
+ p.cu[BLOCK_32x32].psyRdoQuant_2p = psyRdoQuant_c_2<5>;127
p.scanPosLast = scanPosLast_c;128
p.findPosFirstLast = findPosFirstLast_c;129
p.costCoeffNxN = costCoeffNxN_c;130
x265_2.7.tar.gz/source/common/frame.cpp -> x265_2.9.tar.gz/source/common/frame.cpp
Changed
56
1
2
m_addOnDepth = NULL;3
m_addOnCtuInfo = NULL;4
m_addOnPrevChange = NULL;5
+ m_classifyFrame = false;6
}7
8
bool Frame::create(x265_param *param, float* quantOffsets)9
10
m_analysisData.wt = NULL;11
m_analysisData.intraData = NULL;12
m_analysisData.interData = NULL;13
- m_analysis2Pass.analysisFramedata = NULL;14
+ m_analysisData.distortionData = NULL;15
}16
17
- if (m_fencPic->create(param, !!m_param->bCopyPicToFrame) && m_lowres.create(m_fencPic, param->bframes, !!param->rc.aqMode || !!param->bAQMotion, param->rc.qgSize))18
+ if (param->bDynamicRefine)19
+ {20
+ int size = m_param->maxCUDepth * X265_REFINE_INTER_LEVELS;21
+ CHECKED_MALLOC_ZERO(m_classifyRd, uint64_t, size);22
+ CHECKED_MALLOC_ZERO(m_classifyVariance, uint64_t, size);23
+ CHECKED_MALLOC_ZERO(m_classifyCount, uint32_t, size);24
+ }25
+26
+ if (m_fencPic->create(param, !!m_param->bCopyPicToFrame) && m_lowres.create(param, m_fencPic, param->rc.qgSize))27
{28
X265_CHECK((m_reconColCount == NULL), "m_reconColCount was initialized");29
m_numRows = (m_fencPic->m_picHeight + param->maxCUSize - 1) / param->maxCUSize;30
31
32
if (quantOffsets)33
{34
- int32_t cuCount;35
- if (param->rc.qgSize == 8)36
- cuCount = m_lowres.maxBlocksInRowFullRes * m_lowres.maxBlocksInColFullRes;37
- else38
- cuCount = m_lowres.maxBlocksInRow * m_lowres.maxBlocksInCol;39
+ int32_t cuCount = (param->rc.qgSize == 8) ? m_lowres.maxBlocksInRowFullRes * m_lowres.maxBlocksInColFullRes :40
+ m_lowres.maxBlocksInRow * m_lowres.maxBlocksInCol;41
m_quantOffsets = new float[cuCount];42
}43
return true;44
45
}46
m_lowres.destroy();47
X265_FREE(m_rcData);48
+49
+ if (m_param->bDynamicRefine)50
+ {51
+ X265_FREE_ZERO(m_classifyRd);52
+ X265_FREE_ZERO(m_classifyVariance);53
+ X265_FREE_ZERO(m_classifyCount);54
+ }55
}56
x265_2.7.tar.gz/source/common/frame.h -> x265_2.9.tar.gz/source/common/frame.h
Changed
24
1
2
Frame* m_prev;3
x265_param* m_param; // Points to the latest param set for the frame.4
x265_analysis_data m_analysisData;5
- x265_analysis_2Pass m_analysis2Pass;6
RcStats* m_rcData;7
8
Event m_copyMVType;9
10
uint8_t** m_addOnDepth;11
uint8_t** m_addOnCtuInfo;12
int** m_addOnPrevChange;13
+14
+ /* Average feature values of frames being considered for classification */15
+ uint64_t* m_classifyRd;16
+ uint64_t* m_classifyVariance;17
+ uint32_t* m_classifyCount;18
+19
+ bool m_classifyFrame;20
+21
Frame();22
23
bool create(x265_param *param, float* quantOffsets);24
x265_2.7.tar.gz/source/common/framedata.cpp -> x265_2.9.tar.gz/source/common/framedata.cpp
Changed
53
1
2
if (param.rc.bStatWrite)3
m_spsrps = const_cast<RPS*>(sps.spsrps);4
bool isallocated = m_cuMemPool.create(0, param.internalCsp, sps.numCUsInFrame, param);5
+ if (m_param->bDynamicRefine)6
+ {7
+ CHECKED_MALLOC_ZERO(m_cuMemPool.dynRefineRdBlock, uint64_t, MAX_NUM_DYN_REFINE * sps.numCUsInFrame);8
+ CHECKED_MALLOC_ZERO(m_cuMemPool.dynRefCntBlock, uint32_t, MAX_NUM_DYN_REFINE * sps.numCUsInFrame);9
+ CHECKED_MALLOC_ZERO(m_cuMemPool.dynRefVarBlock, uint32_t, MAX_NUM_DYN_REFINE * sps.numCUsInFrame);10
+ }11
if (isallocated)12
+ {13
for (uint32_t ctuAddr = 0; ctuAddr < sps.numCUsInFrame; ctuAddr++)14
+ {15
+ if (m_param->bDynamicRefine)16
+ {17
+ m_picCTU[ctuAddr].m_collectCURd = m_cuMemPool.dynRefineRdBlock + (ctuAddr * MAX_NUM_DYN_REFINE);18
+ m_picCTU[ctuAddr].m_collectCUVariance = m_cuMemPool.dynRefVarBlock + (ctuAddr * MAX_NUM_DYN_REFINE);19
+ m_picCTU[ctuAddr].m_collectCUCount = m_cuMemPool.dynRefCntBlock + (ctuAddr * MAX_NUM_DYN_REFINE);20
+ }21
m_picCTU[ctuAddr].initialize(m_cuMemPool, 0, param, ctuAddr);22
+ }23
+ }24
else25
return false;26
CHECKED_MALLOC_ZERO(m_cuStat, RCStatCU, sps.numCUsInFrame);27
28
{29
memset(m_cuStat, 0, sps.numCUsInFrame * sizeof(*m_cuStat));30
memset(m_rowStat, 0, sps.numCuInHeight * sizeof(*m_rowStat));31
+ if (m_param->bDynamicRefine)32
+ {33
+ memset(m_picCTU->m_collectCURd, 0, MAX_NUM_DYN_REFINE * sizeof(uint64_t));34
+ memset(m_picCTU->m_collectCUVariance, 0, MAX_NUM_DYN_REFINE * sizeof(uint32_t));35
+ memset(m_picCTU->m_collectCUCount, 0, MAX_NUM_DYN_REFINE * sizeof(uint32_t));36
+ }37
}38
39
void FrameData::destroy()40
41
42
m_cuMemPool.destroy();43
44
+ if (m_param->bDynamicRefine)45
+ {46
+ X265_FREE(m_cuMemPool.dynRefineRdBlock);47
+ X265_FREE(m_cuMemPool.dynRefCntBlock);48
+ X265_FREE(m_cuMemPool.dynRefVarBlock);49
+ }50
X265_FREE(m_cuStat);51
X265_FREE(m_rowStat);52
for (int i = 0; i < INTEGRAL_PLANE_NUM; i++)53
x265_2.7.tar.gz/source/common/framedata.h -> x265_2.9.tar.gz/source/common/framedata.h
Changed
61
1
2
uint64_t cntInterPu[NUM_CU_DEPTH][INTER_MODES - 1];3
uint64_t cntMergePu[NUM_CU_DEPTH][INTER_MODES - 1];4
5
+ /* Feature values per row for dynamic refinement */6
+ uint64_t rowRdDyn[MAX_NUM_DYN_REFINE];7
+ uint32_t rowVarDyn[MAX_NUM_DYN_REFINE];8
+ uint32_t rowCntDyn[MAX_NUM_DYN_REFINE];9
+10
FrameStats()11
{12
memset(this, 0, sizeof(FrameStats));13
14
inline CUData* getPicCTU(uint32_t ctuAddr) { return &m_picCTU[ctuAddr]; }15
};16
17
-/* Stores intra analysis data for a single frame. This struct needs better packing */18
-struct analysis_intra_data19
-{20
- uint8_t* depth;21
- uint8_t* modes;22
- char* partSizes;23
- uint8_t* chromaModes;24
-};25
-26
-/* Stores inter analysis data for a single frame */27
-struct analysis_inter_data28
-{29
- int32_t* ref;30
- uint8_t* depth;31
- uint8_t* modes;32
- uint8_t* partSize;33
- uint8_t* mergeFlag;34
- uint8_t* interDir;35
- uint8_t* mvpIdx[2];36
- int8_t* refIdx[2];37
- MV* mv[2];38
- int64_t* sadCost;39
-};40
-41
-struct analysis2PassFrameData42
-{43
- uint8_t* depth;44
- MV* m_mv[2];45
- int* mvpIdx[2];46
- int32_t* ref[2];47
- uint8_t* modes;48
- sse_t* distortion;49
- sse_t* ctuDistortion;50
- double* scaledDistortion;51
- double averageDistortion;52
- double sdDistortion;53
- uint32_t highDistortionCtuCount;54
- uint32_t lowDistortionCtuCount;55
- double* offset;56
- double* threshold;57
-};58
-59
}60
#endif // ifndef X265_FRAMEDATA_H61
x265_2.7.tar.gz/source/common/ipfilter.cpp -> x265_2.9.tar.gz/source/common/ipfilter.cpp
Changed
41
1
2
p.chroma[X265_CSP_I420].pu[CHROMA_420_ ## W ## x ## H].filter_vps = interp_vert_ps_c<4, W, H>; \3
p.chroma[X265_CSP_I420].pu[CHROMA_420_ ## W ## x ## H].filter_vsp = interp_vert_sp_c<4, W, H>; \4
p.chroma[X265_CSP_I420].pu[CHROMA_420_ ## W ## x ## H].filter_vss = interp_vert_ss_c<4, W, H>; \5
- p.chroma[X265_CSP_I420].pu[CHROMA_420_ ## W ## x ## H].p2s = filterPixelToShort_c<W, H>;6
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_ ## W ## x ## H].p2s[NONALIGNED] = filterPixelToShort_c<W, H>;\7
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_ ## W ## x ## H].p2s[ALIGNED] = filterPixelToShort_c<W, H>;8
9
#define CHROMA_422(W, H) \10
p.chroma[X265_CSP_I422].pu[CHROMA_422_ ## W ## x ## H].filter_hpp = interp_horiz_pp_c<4, W, H>; \11
12
p.chroma[X265_CSP_I422].pu[CHROMA_422_ ## W ## x ## H].filter_vps = interp_vert_ps_c<4, W, H>; \13
p.chroma[X265_CSP_I422].pu[CHROMA_422_ ## W ## x ## H].filter_vsp = interp_vert_sp_c<4, W, H>; \14
p.chroma[X265_CSP_I422].pu[CHROMA_422_ ## W ## x ## H].filter_vss = interp_vert_ss_c<4, W, H>; \15
- p.chroma[X265_CSP_I422].pu[CHROMA_422_ ## W ## x ## H].p2s = filterPixelToShort_c<W, H>;16
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_ ## W ## x ## H].p2s[NONALIGNED] = filterPixelToShort_c<W, H>;\17
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_ ## W ## x ## H].p2s[ALIGNED] = filterPixelToShort_c<W, H>;18
19
#define CHROMA_444(W, H) \20
p.chroma[X265_CSP_I444].pu[LUMA_ ## W ## x ## H].filter_hpp = interp_horiz_pp_c<4, W, H>; \21
22
p.chroma[X265_CSP_I444].pu[LUMA_ ## W ## x ## H].filter_vps = interp_vert_ps_c<4, W, H>; \23
p.chroma[X265_CSP_I444].pu[LUMA_ ## W ## x ## H].filter_vsp = interp_vert_sp_c<4, W, H>; \24
p.chroma[X265_CSP_I444].pu[LUMA_ ## W ## x ## H].filter_vss = interp_vert_ss_c<4, W, H>; \25
- p.chroma[X265_CSP_I444].pu[LUMA_ ## W ## x ## H].p2s = filterPixelToShort_c<W, H>;26
+ p.chroma[X265_CSP_I444].pu[LUMA_ ## W ## x ## H].p2s[NONALIGNED] = filterPixelToShort_c<W, H>;\27
+ p.chroma[X265_CSP_I444].pu[LUMA_ ## W ## x ## H].p2s[ALIGNED] = filterPixelToShort_c<W, H>;28
29
#define LUMA(W, H) \30
p.pu[LUMA_ ## W ## x ## H].luma_hpp = interp_horiz_pp_c<8, W, H>; \31
32
p.pu[LUMA_ ## W ## x ## H].luma_vsp = interp_vert_sp_c<8, W, H>; \33
p.pu[LUMA_ ## W ## x ## H].luma_vss = interp_vert_ss_c<8, W, H>; \34
p.pu[LUMA_ ## W ## x ## H].luma_hvpp = interp_hv_pp_c<8, W, H>; \35
- p.pu[LUMA_ ## W ## x ## H].convert_p2s = filterPixelToShort_c<W, H>;36
+ p.pu[LUMA_ ## W ## x ## H].convert_p2s[NONALIGNED] = filterPixelToShort_c<W, H>;\37
+ p.pu[LUMA_ ## W ## x ## H].convert_p2s[ALIGNED] = filterPixelToShort_c<W, H>;38
39
void setupFilterPrimitives_c(EncoderPrimitives& p)40
{41
x265_2.7.tar.gz/source/common/lowres.cpp -> x265_2.9.tar.gz/source/common/lowres.cpp
Changed
66
1
2
3
using namespace X265_NS;4
5
-bool Lowres::create(PicYuv *origPic, int _bframes, bool bAQEnabled, uint32_t qgSize)6
+bool Lowres::create(x265_param* param, PicYuv *origPic, uint32_t qgSize)7
{8
isLowres = true;9
- bframes = _bframes;10
+ bframes = param->bframes;11
width = origPic->m_picWidth / 2;12
lines = origPic->m_picHeight / 2;13
lumaStride = width + 2 * origPic->m_lumaMarginX;14
15
maxBlocksInRowFullRes = maxBlocksInRow * 2;16
maxBlocksInColFullRes = maxBlocksInCol * 2;17
int cuCount = maxBlocksInRow * maxBlocksInCol;18
- int cuCountFullRes;19
- if (qgSize == 8)20
- cuCountFullRes = maxBlocksInRowFullRes * maxBlocksInColFullRes;21
- else22
- cuCountFullRes = cuCount;23
+ int cuCountFullRes = (qgSize > 8) ? cuCount : cuCount << 2;24
25
/* rounding the width to multiple of lowres CU size */26
width = maxBlocksInRow * X265_LOWRES_CU_SIZE;27
28
29
size_t planesize = lumaStride * (lines + 2 * origPic->m_lumaMarginY);30
size_t padoffset = lumaStride * origPic->m_lumaMarginY + origPic->m_lumaMarginX;31
- if (bAQEnabled)32
+ if (!!param->rc.aqMode)33
{34
CHECKED_MALLOC_ZERO(qpAqOffset, double, cuCountFullRes);35
- CHECKED_MALLOC_ZERO(qpAqMotionOffset, double, cuCountFullRes);36
CHECKED_MALLOC_ZERO(invQscaleFactor, int, cuCountFullRes);37
CHECKED_MALLOC_ZERO(qpCuTreeOffset, double, cuCountFullRes);38
- CHECKED_MALLOC_ZERO(blockVariance, uint32_t, cuCountFullRes);39
if (qgSize == 8)40
CHECKED_MALLOC_ZERO(invQscaleFactor8x8, int, cuCount);41
}42
+ if (origPic->m_param->bAQMotion)43
+ CHECKED_MALLOC_ZERO(qpAqMotionOffset, double, cuCountFullRes);44
+ if (origPic->m_param->bDynamicRefine)45
+ CHECKED_MALLOC_ZERO(blockVariance, uint32_t, cuCountFullRes);46
CHECKED_MALLOC(propagateCost, uint16_t, cuCount);47
48
/* allocate lowres buffers */49
50
X265_FREE(lowresMvCosts[1][i]);51
}52
X265_FREE(qpAqOffset);53
- X265_FREE(qpAqMotionOffset);54
X265_FREE(invQscaleFactor);55
X265_FREE(qpCuTreeOffset);56
X265_FREE(propagateCost);57
- X265_FREE(blockVariance);58
X265_FREE(invQscaleFactor8x8);59
+ X265_FREE(qpAqMotionOffset);60
+ X265_FREE(blockVariance);61
}62
-63
// (re) initialize lowres state64
void Lowres::init(PicYuv *origPic, int poc)65
{66
x265_2.7.tar.gz/source/common/lowres.h -> x265_2.9.tar.gz/source/common/lowres.h
Changed
35
1
2
int qmvy = qmv.y + (qmv.y & 1);3
int hpelB = (qmvy & 2) | ((qmvx & 2) >> 1);4
pixel *frefB = lowresPlane[hpelB] + blockOffset + (qmvx >> 2) + (qmvy >> 2) * lumaStride;5
- primitives.pu[LUMA_8x8].pixelavg_pp(buf, outstride, frefA, lumaStride, frefB, lumaStride, 32);6
+ primitives.pu[LUMA_8x8].pixelavg_pp[(outstride % 64 == 0) && (lumaStride % 64 == 0)](buf, outstride, frefA, lumaStride, frefB, lumaStride, 32);7
return buf;8
}9
else10
11
int qmvy = qmv.y + (qmv.y & 1);12
int hpelB = (qmvy & 2) | ((qmvx & 2) >> 1);13
pixel *frefB = lowresPlane[hpelB] + blockOffset + (qmvx >> 2) + (qmvy >> 2) * lumaStride;14
- primitives.pu[LUMA_8x8].pixelavg_pp(subpelbuf, 8, frefA, lumaStride, frefB, lumaStride, 32);15
+ primitives.pu[LUMA_8x8].pixelavg_pp[NONALIGNED](subpelbuf, 8, frefA, lumaStride, frefB, lumaStride, 32);16
return comp(fenc, FENC_STRIDE, subpelbuf, 8);17
}18
else19
20
uint32_t* blockVariance;21
uint64_t wp_ssd[3]; // This is different than SSDY, this is sum(pixel^2) - sum(pixel)^2 for entire frame22
uint64_t wp_sum[3];23
- uint64_t frameVariance;24
25
/* cutree intermediate data */26
uint16_t* propagateCost;27
double weightedCostDelta[X265_BFRAME_MAX + 2];28
ReferencePlanes weightedRef[X265_BFRAME_MAX + 2];29
-30
- bool create(PicYuv *origPic, int _bframes, bool bAqEnabled, uint32_t qgSize);31
+ bool create(x265_param* param, PicYuv *origPic, uint32_t qgSize);32
void destroy();33
void init(PicYuv *origPic, int poc);34
};35
x265_2.7.tar.gz/source/common/param.cpp -> x265_2.9.tar.gz/source/common/param.cpp
Changed
224
1
2
memset(param, 0, sizeof(x265_param));3
4
/* Applying default values to all elements in the param structure */5
- param->cpuid = X265_NS::cpu_detect();6
+ param->cpuid = X265_NS::cpu_detect(false);7
param->bEnableWavefront = 1;8
param->frameNumThreads = 0;9
10
11
param->bEmitHRDSEI = 0;12
param->bEmitInfoSEI = 1;13
param->bEmitHDRSEI = 0;14
+ param->bEmitIDRRecoverySEI = 0;15
16
/* CU definitions */17
param->maxCUSize = 64;18
19
param->lookaheadThreads = 0;20
param->scenecutBias = 5.0;21
param->radl = 0;22
+ param->chunkStart = 0;23
+ param->chunkEnd = 0;24
+25
/* Intra Coding Tools */26
param->bEnableConstrainedIntra = 0;27
param->bEnableStrongIntraSmoothing = 1;28
29
param->bEnableSAO = 1;30
param->bSaoNonDeblocked = 0;31
param->bLimitSAO = 0;32
+33
/* Coding Quality */34
param->cbQpOffset = 0;35
param->crQpOffset = 0;36
37
param->scaleFactor = 0;38
param->intraRefine = 0;39
param->interRefine = 0;40
+ param->bDynamicRefine = 0;41
param->mvRefine = 0;42
param->bUseAnalysisFile = 1;43
param->csvfpt = NULL;44
param->forceFlush = 0;45
param->bDisableLookahead = 0;46
param->bCopyPicToFrame = 1;47
+ param->maxAUSizeFactor = 1;48
+ param->naluFile = NULL;49
50
/* DCT Approximations */51
param->bLowPassDct = 0;52
param->bMVType = 0;53
+ param->bSingleSeiNal = 0;54
+55
+ /* SEI messages */56
+ param->preferredTransferCharacteristics = -1;57
+ param->pictureStructure = -1;58
}59
60
int x265_param_default_preset(x265_param* param, const char* preset, const char* tune)61
62
if (0) ;63
OPT("asm")64
{65
+#if X265_ARCH_X8666
+ if (!strcasecmp(value, "avx512"))67
+ {68
+ p->cpuid = X265_NS::cpu_detect(true);69
+ if (!(p->cpuid & X265_CPU_AVX512))70
+ x265_log(p, X265_LOG_WARNING, "AVX512 is not supported\n");71
+ }72
+ else73
+ {74
+ if (bValueWasNull)75
+ p->cpuid = atobool(value);76
+ else77
+ p->cpuid = parseCpuName(value, bError, false);78
+ }79
+#else80
if (bValueWasNull)81
p->cpuid = atobool(value);82
else83
- p->cpuid = parseCpuName(value, bError);84
+ p->cpuid = parseCpuName(value, bError, false);85
+#endif86
}87
OPT("fps")88
{89
90
OPT("limit-sao") p->bLimitSAO = atobool(value);91
OPT("dhdr10-info") p->toneMapFile = strdup(value);92
OPT("dhdr10-opt") p->bDhdr10opt = atobool(value);93
+ OPT("idr-recovery-sei") p->bEmitIDRRecoverySEI = atobool(value);94
OPT("const-vbv") p->rc.bEnableConstVbv = atobool(value);95
OPT("ctu-info") p->bCTUInfo = atoi(value);96
OPT("scale-factor") p->scaleFactor = atoi(value);97
98
OPT("refine-mv")p->mvRefine = atobool(value);99
OPT("force-flush")p->forceFlush = atoi(value);100
OPT("splitrd-skip") p->bEnableSplitRdSkip = atobool(value);101
- OPT("lowpass-dct") p->bLowPassDct = atobool(value);102
+ OPT("lowpass-dct") p->bLowPassDct = atobool(value);103
OPT("vbv-end") p->vbvBufferEnd = atof(value);104
OPT("vbv-end-fr-adj") p->vbvEndFrameAdjust = atof(value);105
OPT("copy-pic") p->bCopyPicToFrame = atobool(value);106
107
{108
bError = true;109
}110
- }111
+ }112
OPT("gop-lookahead") p->gopLookahead = atoi(value);113
OPT("analysis-save") p->analysisSave = strdup(value);114
OPT("analysis-load") p->analysisLoad = strdup(value);115
OPT("radl") p->radl = atoi(value);116
+ OPT("max-ausize-factor") p->maxAUSizeFactor = atof(value);117
+ OPT("dynamic-refine") p->bDynamicRefine = atobool(value);118
+ OPT("single-sei") p->bSingleSeiNal = atobool(value);119
+ OPT("atc-sei") p->preferredTransferCharacteristics = atoi(value);120
+ OPT("pic-struct") p->pictureStructure = atoi(value);121
+ OPT("chunk-start") p->chunkStart = atoi(value);122
+ OPT("chunk-end") p->chunkEnd = atoi(value);123
+ OPT("nalu-file") p->naluFile = strdup(value);124
else125
return X265_PARAM_BAD_NAME;126
}127
128
* false || no - disabled129
* integer bitmap value130
* comma separated list of SIMD names, eg: SSE4.1,XOP */131
-int parseCpuName(const char* value, bool& bError)132
+int parseCpuName(const char* value, bool& bError, bool bEnableavx512)133
{134
if (!value)135
{136
137
if (isdigit(value[0]))138
cpu = x265_atoi(value, bError);139
else140
- cpu = !strcmp(value, "auto") || x265_atobool(value, bError) ? X265_NS::cpu_detect() : 0;141
+ cpu = !strcmp(value, "auto") || x265_atobool(value, bError) ? X265_NS::cpu_detect(bEnableavx512) : 0;142
143
if (bError)144
{145
146
"Supported values for bCTUInfo are 0, 1, 2, 4, 6");147
CHECK(param->interRefine > 3 || param->interRefine < 0,148
"Invalid refine-inter value, refine-inter levels 0 to 3 supported");149
- CHECK(param->intraRefine > 3 || param->intraRefine < 0,150
+ CHECK(param->intraRefine > 4 || param->intraRefine < 0,151
"Invalid refine-intra value, refine-intra levels 0 to 3 supported");152
+ CHECK(param->maxAUSizeFactor < 0.5 || param->maxAUSizeFactor > 1.0,153
+ "Supported factor for controlling max AU size is from 0.5 to 1");154
#if !X86_64155
CHECK(param->searchMethod == X265_SEA && (param->sourceWidth > 840 || param->sourceHeight > 480),156
"SEA motion search does not support resolutions greater than 480p in 32 bit build");157
158
if (param->masteringDisplayColorVolume || param->maxFALL || param->maxCLL)159
param->bEmitHDRSEI = 1;160
161
+ bool isSingleSEI = (param->bRepeatHeaders162
+ || param->bEmitHRDSEI163
+ || param->bEmitInfoSEI164
+ || param->bEmitHDRSEI165
+ || param->bEmitIDRRecoverySEI166
+ || !!param->interlaceMode167
+ || param->preferredTransferCharacteristics > 1168
+ || param->toneMapFile169
+ || param->naluFile);170
+171
+ if (!isSingleSEI && param->bSingleSeiNal)172
+ {173
+ param->bSingleSeiNal = 0;174
+ x265_log(param, X265_LOG_WARNING, "None of the SEI messages are enabled. Disabling Single SEI NAL\n");175
+ }176
return check_failed;177
}178
179
180
TOOLVAL(param->bCTUInfo, "ctu-info=%d");181
if (param->bMVType == AVC_INFO)182
TOOLOPT(param->bMVType, "refine-mv-type=avc");183
+ TOOLOPT(param->bDynamicRefine, "dynamic-refine");184
if (param->maxSlices > 1)185
TOOLVAL(param->maxSlices, "slices=%d");186
if (param->bEnableLoopFilter)187
188
TOOLOPT(!param->bSaoNonDeblocked && param->bEnableSAO, "sao");189
TOOLOPT(param->rc.bStatWrite, "stats-write");190
TOOLOPT(param->rc.bStatRead, "stats-read");191
+ TOOLOPT(param->bSingleSeiNal, "single-sei");192
#if ENABLE_HDR10_PLUS193
TOOLOPT(param->toneMapFile != NULL, "dhdr10-info");194
#endif195
196
s += sprintf(s, " input-res=%dx%d", p->sourceWidth - padx, p->sourceHeight - pady);197
s += sprintf(s, " interlace=%d", p->interlaceMode);198
s += sprintf(s, " total-frames=%d", p->totalFrames);199
+ if (p->chunkStart)200
+ s += sprintf(s, " chunk-start=%d", p->chunkStart);201
+ if (p->chunkEnd)202
+ s += sprintf(s, " chunk-end=%d", p->chunkEnd);203
s += sprintf(s, " level-idc=%d", p->levelIdc);204
s += sprintf(s, " high-tier=%d", p->bHighTier);205
s += sprintf(s, " uhd-bd=%d", p->uhdBluray);206
207
BOOL(p->bEmitHDRSEI, "hdr");208
BOOL(p->bHDROpt, "hdr-opt");209
BOOL(p->bDhdr10opt, "dhdr10-opt");210
+ BOOL(p->bEmitIDRRecoverySEI, "idr-recovery-sei");211
if (p->analysisSave)212
s += sprintf(s, " analysis-save");213
if (p->analysisLoad)214
215
BOOL(p->bLowPassDct, "lowpass-dct");216
s += sprintf(s, " refine-mv-type=%d", p->bMVType);217
s += sprintf(s, " copy-pic=%d", p->bCopyPicToFrame);218
+ s += sprintf(s, " max-ausize-factor=%.1f", p->maxAUSizeFactor);219
+ BOOL(p->bDynamicRefine, "dynamic-refine");220
+ BOOL(p->bSingleSeiNal, "single-sei");221
#undef BOOL222
return buf;223
}224
x265_2.7.tar.gz/source/common/param.h -> x265_2.9.tar.gz/source/common/param.h
Changed
10
1
2
char* x265_param2string(x265_param *param, int padx, int pady);3
int x265_atoi(const char *str, bool& bError);4
double x265_atof(const char *str, bool& bError);5
-int parseCpuName(const char *value, bool& bError);6
+int parseCpuName(const char *value, bool& bError, bool bEnableavx512);7
void setParamAspectRatio(x265_param *p, int width, int height);8
void getParamAspectRatio(x265_param *p, int& width, int& height);9
bool parseLambdaFile(x265_param *param);10
x265_2.7.tar.gz/source/common/picyuv.cpp -> x265_2.9.tar.gz/source/common/picyuv.cpp
Changed
21
1
2
pixel *uPic = m_picOrg[1];3
pixel *vPic = m_picOrg[2];4
5
+ if(param.minLuma != 0 || param.maxLuma != PIXEL_MAX)6
+ {7
+ for (int r = 0; r < height; r++)8
+ {9
+ for (int c = 0; c < width; c++)10
+ {11
+ yPic[c] = X265_MIN(yPic[c], (pixel)param.maxLuma);12
+ yPic[c] = X265_MAX(yPic[c], (pixel)param.minLuma);13
+ }14
+ yPic += m_stride;15
+ }16
+ }17
+ yPic = m_picOrg[0];18
if (param.csvLogLevel >= 2 || param.maxCLL || param.maxFALL)19
{20
for (int r = 0; r < height; r++)21
x265_2.7.tar.gz/source/common/picyuv.h -> x265_2.9.tar.gz/source/common/picyuv.h
Changed
9
1
2
pixel m_maxChromaVLevel;3
pixel m_minChromaVLevel;4
double m_avgChromaVLevel;5
+ double m_vmafScore;6
x265_param *m_param;7
8
PicYuv();9
x265_2.7.tar.gz/source/common/pixel.cpp -> x265_2.9.tar.gz/source/common/pixel.cpp
Changed
102
1
2
static void cuTreeFix8Pack(uint16_t *dst, double *src, int count)3
{4
for (int i = 0; i < count; i++)5
- dst[i] = (uint16_t)(src[i] * 256.0);6
+ dst[i] = (uint16_t)(int16_t)(src[i] * 256.0);7
}8
9
static void cuTreeFix8Unpack(double *dst, uint16_t *src, int count)10
11
{12
#define LUMA_PU(W, H) \13
p.pu[LUMA_ ## W ## x ## H].copy_pp = blockcopy_pp_c<W, H>; \14
- p.pu[LUMA_ ## W ## x ## H].addAvg = addAvg<W, H>; \15
+ p.pu[LUMA_ ## W ## x ## H].addAvg[NONALIGNED] = addAvg<W, H>; \16
+ p.pu[LUMA_ ## W ## x ## H].addAvg[ALIGNED] = addAvg<W, H>; \17
p.pu[LUMA_ ## W ## x ## H].sad = sad<W, H>; \18
p.pu[LUMA_ ## W ## x ## H].sad_x3 = sad_x3<W, H>; \19
p.pu[LUMA_ ## W ## x ## H].sad_x4 = sad_x4<W, H>; \20
- p.pu[LUMA_ ## W ## x ## H].pixelavg_pp = pixelavg_pp<W, H>;21
-22
+ p.pu[LUMA_ ## W ## x ## H].pixelavg_pp[NONALIGNED] = pixelavg_pp<W, H>; \23
+ p.pu[LUMA_ ## W ## x ## H].pixelavg_pp[ALIGNED] = pixelavg_pp<W, H>;24
#define LUMA_CU(W, H) \25
p.cu[BLOCK_ ## W ## x ## H].sub_ps = pixel_sub_ps_c<W, H>; \26
- p.cu[BLOCK_ ## W ## x ## H].add_ps = pixel_add_ps_c<W, H>; \27
+ p.cu[BLOCK_ ## W ## x ## H].add_ps[NONALIGNED] = pixel_add_ps_c<W, H>; \28
+ p.cu[BLOCK_ ## W ## x ## H].add_ps[ALIGNED] = pixel_add_ps_c<W, H>; \29
p.cu[BLOCK_ ## W ## x ## H].copy_sp = blockcopy_sp_c<W, H>; \30
p.cu[BLOCK_ ## W ## x ## H].copy_ps = blockcopy_ps_c<W, H>; \31
p.cu[BLOCK_ ## W ## x ## H].copy_ss = blockcopy_ss_c<W, H>; \32
- p.cu[BLOCK_ ## W ## x ## H].blockfill_s = blockfill_s_c<W>; \33
+ p.cu[BLOCK_ ## W ## x ## H].blockfill_s[NONALIGNED] = blockfill_s_c<W>; \34
+ p.cu[BLOCK_ ## W ## x ## H].blockfill_s[ALIGNED] = blockfill_s_c<W>; \35
p.cu[BLOCK_ ## W ## x ## H].cpy2Dto1D_shl = cpy2Dto1D_shl<W>; \36
p.cu[BLOCK_ ## W ## x ## H].cpy2Dto1D_shr = cpy2Dto1D_shr<W>; \37
- p.cu[BLOCK_ ## W ## x ## H].cpy1Dto2D_shl = cpy1Dto2D_shl<W>; \38
+ p.cu[BLOCK_ ## W ## x ## H].cpy1Dto2D_shl[NONALIGNED] = cpy1Dto2D_shl<W>; \39
+ p.cu[BLOCK_ ## W ## x ## H].cpy1Dto2D_shl[ALIGNED] = cpy1Dto2D_shl<W>; \40
p.cu[BLOCK_ ## W ## x ## H].cpy1Dto2D_shr = cpy1Dto2D_shr<W>; \41
p.cu[BLOCK_ ## W ## x ## H].psy_cost_pp = psyCost_pp<BLOCK_ ## W ## x ## H>; \42
p.cu[BLOCK_ ## W ## x ## H].transpose = transpose<W>; \43
- p.cu[BLOCK_ ## W ## x ## H].ssd_s = pixel_ssd_s_c<W>; \44
+ p.cu[BLOCK_ ## W ## x ## H].ssd_s[NONALIGNED] = pixel_ssd_s_c<W>; \45
+ p.cu[BLOCK_ ## W ## x ## H].ssd_s[ALIGNED] = pixel_ssd_s_c<W>; \46
p.cu[BLOCK_ ## W ## x ## H].var = pixel_var<W>; \47
- p.cu[BLOCK_ ## W ## x ## H].calcresidual = getResidual<W>; \48
+ p.cu[BLOCK_ ## W ## x ## H].calcresidual[NONALIGNED] = getResidual<W>; \49
+ p.cu[BLOCK_ ## W ## x ## H].calcresidual[ALIGNED] = getResidual<W>; \50
p.cu[BLOCK_ ## W ## x ## H].sse_pp = sse<W, H, pixel, pixel>; \51
p.cu[BLOCK_ ## W ## x ## H].sse_ss = sse<W, H, int16_t, int16_t>;52
53
54
p.cu[BLOCK_64x64].sa8d = sa8d16<64, 64>;55
56
#define CHROMA_PU_420(W, H) \57
- p.chroma[X265_CSP_I420].pu[CHROMA_420_ ## W ## x ## H].addAvg = addAvg<W, H>; \58
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_ ## W ## x ## H].addAvg[NONALIGNED] = addAvg<W, H>; \59
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_ ## W ## x ## H].addAvg[ALIGNED] = addAvg<W, H>; \60
p.chroma[X265_CSP_I420].pu[CHROMA_420_ ## W ## x ## H].copy_pp = blockcopy_pp_c<W, H>; \61
62
CHROMA_PU_420(2, 2);63
64
p.chroma[X265_CSP_I420].cu[BLOCK_420_ ## W ## x ## H].copy_ps = blockcopy_ps_c<W, H>; \65
p.chroma[X265_CSP_I420].cu[BLOCK_420_ ## W ## x ## H].copy_ss = blockcopy_ss_c<W, H>; \66
p.chroma[X265_CSP_I420].cu[BLOCK_420_ ## W ## x ## H].sub_ps = pixel_sub_ps_c<W, H>; \67
- p.chroma[X265_CSP_I420].cu[BLOCK_420_ ## W ## x ## H].add_ps = pixel_add_ps_c<W, H>;68
+ p.chroma[X265_CSP_I420].cu[BLOCK_420_ ## W ## x ## H].add_ps[NONALIGNED] = pixel_add_ps_c<W, H>; \69
+ p.chroma[X265_CSP_I420].cu[BLOCK_420_ ## W ## x ## H].add_ps[ALIGNED] = pixel_add_ps_c<W, H>;70
71
CHROMA_CU_420(2, 2)72
CHROMA_CU_420(4, 4)73
74
p.chroma[X265_CSP_I420].cu[BLOCK_64x64].sa8d = sa8d16<32, 32>;75
76
#define CHROMA_PU_422(W, H) \77
- p.chroma[X265_CSP_I422].pu[CHROMA_422_ ## W ## x ## H].addAvg = addAvg<W, H>; \78
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_ ## W ## x ## H].addAvg[NONALIGNED] = addAvg<W, H>; \79
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_ ## W ## x ## H].addAvg[ALIGNED] = addAvg<W, H>; \80
p.chroma[X265_CSP_I422].pu[CHROMA_422_ ## W ## x ## H].copy_pp = blockcopy_pp_c<W, H>; \81
82
CHROMA_PU_422(2, 4);83
84
p.chroma[X265_CSP_I422].cu[BLOCK_422_ ## W ## x ## H].copy_ps = blockcopy_ps_c<W, H>; \85
p.chroma[X265_CSP_I422].cu[BLOCK_422_ ## W ## x ## H].copy_ss = blockcopy_ss_c<W, H>; \86
p.chroma[X265_CSP_I422].cu[BLOCK_422_ ## W ## x ## H].sub_ps = pixel_sub_ps_c<W, H>; \87
- p.chroma[X265_CSP_I422].cu[BLOCK_422_ ## W ## x ## H].add_ps = pixel_add_ps_c<W, H>;88
+ p.chroma[X265_CSP_I422].cu[BLOCK_422_ ## W ## x ## H].add_ps[NONALIGNED] = pixel_add_ps_c<W, H>; \89
+ p.chroma[X265_CSP_I422].cu[BLOCK_422_ ## W ## x ## H].add_ps[ALIGNED] = pixel_add_ps_c<W, H>;90
91
CHROMA_CU_422(2, 4)92
CHROMA_CU_422(4, 8)93
94
p.weight_pp = weight_pp_c;95
p.weight_sp = weight_sp_c;96
97
- p.scale1D_128to64 = scale1D_128to64;98
+ p.scale1D_128to64[NONALIGNED] = p.scale1D_128to64[ALIGNED] = scale1D_128to64;99
p.scale2D_64to32 = scale2D_64to32;100
p.frameInitLowres = frame_init_lowres_core;101
p.ssim_4x4x2_core = ssim_4x4x2_core;102
x265_2.7.tar.gz/source/common/predict.cpp -> x265_2.9.tar.gz/source/common/predict.cpp
Changed
72
1
2
MV mv0 = cu.m_mv[0][pu.puAbsPartIdx];3
cu.clipMv(mv0);4
5
- if (cu.m_slice->m_pps->bUseWeightPred && wp0->bPresentFlag)6
+ if (cu.m_slice->m_pps->bUseWeightPred && wp0->wtPresent)7
{8
for (int plane = 0; plane < (bChroma ? 3 : 1); plane++)9
{10
11
pwp0 = refIdx0 >= 0 ? cu.m_slice->m_weightPredTable[0][refIdx0] : NULL;12
pwp1 = refIdx1 >= 0 ? cu.m_slice->m_weightPredTable[1][refIdx1] : NULL;13
14
- if (pwp0 && pwp1 && (pwp0->bPresentFlag || pwp1->bPresentFlag))15
+ if (pwp0 && pwp1 && (pwp0->wtPresent || pwp1->wtPresent))16
{17
/* biprediction weighting */18
for (int plane = 0; plane < (bChroma ? 3 : 1); plane++)19
20
predInterChromaShort(pu, m_predShortYuv[1], *cu.m_slice->m_refReconPicList[1][refIdx1], mv1);21
}22
23
- if (pwp0 && pwp1 && (pwp0->bPresentFlag || pwp1->bPresentFlag))24
+ if (pwp0 && pwp1 && (pwp0->wtPresent || pwp1->wtPresent))25
addWeightBi(pu, predYuv, m_predShortYuv[0], m_predShortYuv[1], wv0, wv1, bLuma, bChroma);26
else27
predYuv.addAvg(m_predShortYuv[0], m_predShortYuv[1], pu.puAbsPartIdx, pu.width, pu.height, bLuma, bChroma);28
29
MV mv0 = cu.m_mv[0][pu.puAbsPartIdx];30
cu.clipMv(mv0);31
32
- if (pwp0 && pwp0->bPresentFlag)33
+ if (pwp0 && pwp0->wtPresent)34
{35
ShortYuv& shortYuv = m_predShortYuv[0];36
37
38
/* uniprediction to L1 */39
X265_CHECK(refIdx1 >= 0, "refidx1 was not positive\n");40
41
- if (pwp1 && pwp1->bPresentFlag)42
+ if (pwp1 && pwp1->wtPresent)43
{44
ShortYuv& shortYuv = m_predShortYuv[0];45
46
47
int yFrac = mv.y & 3;48
49
if (!(yFrac | xFrac))50
- primitives.pu[partEnum].convert_p2s(src, srcStride, dst, dstStride);51
+ {52
+ bool srcbufferAlignCheck = (refPic.m_cuOffsetY[pu.ctuAddr] + refPic.m_buOffsetY[pu.cuAbsPartIdx + pu.puAbsPartIdx] + srcOffset) % 64 == 0;53
+ bool dstbufferAlignCheck = (dstSYuv.getAddrOffset(pu.puAbsPartIdx, dstSYuv.m_size) % 64) == 0;54
+ primitives.pu[partEnum].convert_p2s[srcStride % 64 == 0 && dstStride % 64 == 0 && srcbufferAlignCheck && dstbufferAlignCheck](src, srcStride, dst, dstStride);55
+ }56
else if (!yFrac)57
primitives.pu[partEnum].luma_hps(src, srcStride, dst, dstStride, xFrac, 0);58
else if (!xFrac)59
60
61
if (!(yFrac | xFrac))62
{63
- primitives.chroma[m_csp].pu[partEnum].p2s(refCb, refStride, dstCb, dstStride);64
- primitives.chroma[m_csp].pu[partEnum].p2s(refCr, refStride, dstCr, dstStride);65
+ bool srcbufferAlignCheckC = (refPic.m_cuOffsetC[pu.ctuAddr] + refPic.m_buOffsetC[pu.cuAbsPartIdx + pu.puAbsPartIdx] + refOffset) % 64 == 0;66
+ bool dstbufferAlignCheckC = dstSYuv.getChromaAddrOffset(pu.puAbsPartIdx) % 64 == 0;67
+ primitives.chroma[m_csp].pu[partEnum].p2s[refStride % 64 == 0 && dstStride % 64 == 0 && srcbufferAlignCheckC && dstbufferAlignCheckC](refCb, refStride, dstCb, dstStride);68
+ primitives.chroma[m_csp].pu[partEnum].p2s[refStride % 64 == 0 && dstStride % 64 == 0 && srcbufferAlignCheckC && dstbufferAlignCheckC](refCr, refStride, dstCr, dstStride);69
}70
else if (!yFrac)71
{72
x265_2.7.tar.gz/source/common/primitives.cpp -> x265_2.9.tar.gz/source/common/primitives.cpp
Changed
25
1
2
for (int i = 0; i < NUM_PU_SIZES; i++)3
{4
p.chroma[X265_CSP_I444].pu[i].copy_pp = p.pu[i].copy_pp;5
- p.chroma[X265_CSP_I444].pu[i].addAvg = p.pu[i].addAvg;6
+ p.chroma[X265_CSP_I444].pu[i].addAvg[NONALIGNED] = p.pu[i].addAvg[NONALIGNED];7
+ p.chroma[X265_CSP_I444].pu[i].addAvg[ALIGNED] = p.pu[i].addAvg[ALIGNED];8
p.chroma[X265_CSP_I444].pu[i].satd = p.pu[i].satd;9
- p.chroma[X265_CSP_I444].pu[i].p2s = p.pu[i].convert_p2s;10
+ p.chroma[X265_CSP_I444].pu[i].p2s[NONALIGNED] = p.pu[i].convert_p2s[NONALIGNED];11
+ p.chroma[X265_CSP_I444].pu[i].p2s[ALIGNED] = p.pu[i].convert_p2s[ALIGNED];12
}13
14
for (int i = 0; i < NUM_CU_SIZES; i++)15
16
p.chroma[X265_CSP_I444].cu[i].sa8d = p.cu[i].sa8d;17
p.chroma[X265_CSP_I444].cu[i].sse_pp = p.cu[i].sse_pp;18
p.chroma[X265_CSP_I444].cu[i].sub_ps = p.cu[i].sub_ps;19
- p.chroma[X265_CSP_I444].cu[i].add_ps = p.cu[i].add_ps;20
+ p.chroma[X265_CSP_I444].cu[i].add_ps[NONALIGNED] = p.cu[i].add_ps[NONALIGNED];21
+ p.chroma[X265_CSP_I444].cu[i].add_ps[ALIGNED] = p.cu[i].add_ps[ALIGNED];22
p.chroma[X265_CSP_I444].cu[i].copy_ps = p.cu[i].copy_ps;23
p.chroma[X265_CSP_I444].cu[i].copy_sp = p.cu[i].copy_sp;24
p.chroma[X265_CSP_I444].cu[i].copy_ss = p.cu[i].copy_ss;25
x265_2.7.tar.gz/source/common/primitives.h -> x265_2.9.tar.gz/source/common/primitives.h
Changed
117
1
2
NUM_CU_SIZES3
};4
5
+enum AlignPrimitive6
+{7
+ NONALIGNED,8
+ ALIGNED,9
+ NUM_ALIGNMENT_TYPES10
+};11
+12
enum { NUM_TR_SIZE = 4 }; // TU are 4x4, 8x8, 16x16, and 32x3213
14
15
16
17
typedef void (*integralv_t)(uint32_t *sum, intptr_t stride);18
typedef void (*integralh_t)(uint32_t *sum, pixel *pix, intptr_t stride);19
-20
+typedef void(*nonPsyRdoQuant_t)(int16_t *m_resiDctCoeff, int64_t *costUncoded, int64_t *totalUncodedCost, int64_t *totalRdCost, uint32_t blkPos);21
+typedef void(*psyRdoQuant_t)(int16_t *m_resiDctCoeff, int16_t *m_fencDctCoeff, int64_t *costUncoded, int64_t *totalUncodedCost, int64_t *totalRdCost, int64_t *psyScale, uint32_t blkPos);22
+typedef void(*psyRdoQuant_t1)(int16_t *m_resiDctCoeff, int64_t *costUncoded, int64_t *totalUncodedCost, int64_t *totalRdCost,uint32_t blkPos);23
+typedef void(*psyRdoQuant_t2)(int16_t *m_resiDctCoeff, int16_t *m_fencDctCoeff, int64_t *costUncoded, int64_t *totalUncodedCost, int64_t *totalRdCost, int64_t *psyScale, uint32_t blkPos);24
/* Function pointers to optimized encoder primitives. Each pointer can reference25
* either an assembly routine, a SIMD intrinsic primitive, or a C function */26
struct EncoderPrimitives27
28
filter_sp_t luma_vsp;29
filter_ss_t luma_vss;30
filter_hv_pp_t luma_hvpp; // combines hps + vsp31
-32
- pixelavg_pp_t pixelavg_pp; // quick bidir using pixels (borrowed from x264)33
- addAvg_t addAvg; // bidir motion compensation, uses 16bit values34
-35
+ pixelavg_pp_t pixelavg_pp[NUM_ALIGNMENT_TYPES]; // quick bidir using pixels (borrowed from x264)36
+ addAvg_t addAvg[NUM_ALIGNMENT_TYPES]; // bidir motion compensation, uses 16bit values37
copy_pp_t copy_pp;38
- filter_p2s_t convert_p2s;39
+ filter_p2s_t convert_p2s[NUM_ALIGNMENT_TYPES];40
}41
pu[NUM_PU_SIZES];42
43
44
dct_t standard_dct; // original dct function, used by lowpass_dct45
dct_t lowpass_dct; // lowpass dct approximation46
47
- calcresidual_t calcresidual;48
+ calcresidual_t calcresidual[NUM_ALIGNMENT_TYPES];49
pixel_sub_ps_t sub_ps;50
- pixel_add_ps_t add_ps;51
- blockfill_s_t blockfill_s; // block fill, for DC transforms52
+ pixel_add_ps_t add_ps[NUM_ALIGNMENT_TYPES];53
+ blockfill_s_t blockfill_s[NUM_ALIGNMENT_TYPES]; // block fill, for DC transforms54
copy_cnt_t copy_cnt; // copy coeff while counting non-zero55
count_nonzero_t count_nonzero;56
cpy2Dto1D_shl_t cpy2Dto1D_shl;57
cpy2Dto1D_shr_t cpy2Dto1D_shr;58
- cpy1Dto2D_shl_t cpy1Dto2D_shl;59
+ cpy1Dto2D_shl_t cpy1Dto2D_shl[NUM_ALIGNMENT_TYPES];60
cpy1Dto2D_shr_t cpy1Dto2D_shr;61
-62
copy_sp_t copy_sp;63
copy_ps_t copy_ps;64
copy_ss_t copy_ss;65
66
pixel_sse_t sse_pp; // Sum of Square Error (pixel, pixel) fenc alignment not assumed67
pixel_sse_ss_t sse_ss; // Sum of Square Error (short, short) fenc alignment not assumed68
pixelcmp_t psy_cost_pp; // difference in AC energy between two pixel blocks69
- pixel_ssd_s_t ssd_s; // Sum of Square Error (residual coeff to self)70
+ pixel_ssd_s_t ssd_s[NUM_ALIGNMENT_TYPES]; // Sum of Square Error (residual coeff to self)71
pixelcmp_t sa8d; // Sum of Transformed Differences (8x8 Hadamard), uses satd for 4x4 intra TU72
-73
transpose_t transpose; // transpose pixel block; for use with intra all-angs74
intra_allangs_t intra_pred_allangs;75
intra_filter_t intra_filter;76
intra_pred_t intra_pred[NUM_INTRA_MODE];77
+ nonPsyRdoQuant_t nonPsyRdoQuant;78
+ psyRdoQuant_t psyRdoQuant;79
+ psyRdoQuant_t1 psyRdoQuant_1p;80
+ psyRdoQuant_t2 psyRdoQuant_2p;81
}82
cu[NUM_CU_SIZES];83
-84
/* These remaining primitives work on either fixed block sizes or take85
* block dimensions as arguments and thus do not belong in either the PU or86
* the CU arrays */87
88
dequant_scaling_t dequant_scaling;89
dequant_normal_t dequant_normal;90
denoiseDct_t denoiseDct;91
- scale1D_t scale1D_128to64;92
+ scale1D_t scale1D_128to64[NUM_ALIGNMENT_TYPES];93
scale2D_t scale2D_64to32;94
95
ssim_4x4x2_core_t ssim_4x4x2_core;96
97
filter_ss_t filter_vss;98
filter_pp_t filter_hpp;99
filter_hps_t filter_hps;100
- addAvg_t addAvg;101
+ addAvg_t addAvg[NUM_ALIGNMENT_TYPES];102
copy_pp_t copy_pp;103
- filter_p2s_t p2s;104
+ filter_p2s_t p2s[NUM_ALIGNMENT_TYPES];105
106
}107
pu[NUM_PU_SIZES];108
109
pixelcmp_t sa8d; // if chroma CU is not multiple of 8x8, will use satd110
pixel_sse_t sse_pp;111
pixel_sub_ps_t sub_ps;112
- pixel_add_ps_t add_ps;113
+ pixel_add_ps_t add_ps[NUM_ALIGNMENT_TYPES];114
115
copy_ps_t copy_ps;116
copy_sp_t copy_sp;117
x265_2.7.tar.gz/source/common/quant.cpp -> x265_2.9.tar.gz/source/common/quant.cpp
Changed
163
1
2
uint32_t log2TrSize, TextType ttype, bool bIntra, bool useTransformSkip, uint32_t numSig)3
{4
const uint32_t sizeIdx = log2TrSize - 2;5
-6
if (cu.m_tqBypass[0])7
{8
- primitives.cu[sizeIdx].cpy1Dto2D_shl(residual, coeff, resiStride, 0);9
+ primitives.cu[sizeIdx].cpy1Dto2D_shl[resiStride % 64 == 0](residual, coeff, resiStride, 0);10
return;11
}12
-13
// Values need to pass as input parameter in dequant14
int rem = m_qpParam[ttype].rem;15
int per = m_qpParam[ttype].per;16
17
if (transformShift > 0)18
primitives.cu[sizeIdx].cpy1Dto2D_shr(residual, m_resiDctCoeff, resiStride, transformShift);19
else20
- primitives.cu[sizeIdx].cpy1Dto2D_shl(residual, m_resiDctCoeff, resiStride, -transformShift);21
+ primitives.cu[sizeIdx].cpy1Dto2D_shl[resiStride % 64 == 0](residual, m_resiDctCoeff, resiStride, -transformShift);22
#endif23
}24
else25
26
const int add_2nd = 1 << (shift_2nd - 1);27
28
int dc_val = (((m_resiDctCoeff[0] * (64 >> 6) + add_1st) >> shift_1st) * (64 >> 3) + add_2nd) >> shift_2nd;29
- primitives.cu[sizeIdx].blockfill_s(residual, resiStride, (int16_t)dc_val);30
+ primitives.cu[sizeIdx].blockfill_s[resiStride % 64 == 0](residual, resiStride, (int16_t)dc_val);31
return;32
}33
34
35
X265_CHECK((int)numSig == primitives.cu[log2TrSize - 2].count_nonzero(dstCoeff), "numSig differ\n");36
if (!numSig)37
return 0;38
-39
const uint32_t trSize = 1 << log2TrSize;40
int64_t lambda2 = m_qpParam[ttype].lambda2;41
- const int64_t psyScale = ((int64_t)m_psyRdoqScale * m_qpParam[ttype].lambda);42
-43
+ int64_t psyScale = ((int64_t)m_psyRdoqScale * m_qpParam[ttype].lambda);44
/* unquant constants for measuring distortion. Scaling list quant coefficients have a (1 << 4)45
* scale applied that must be removed during unquant. Note that in real dequant there is clipping46
* at several stages. We skip the clipping for simplicity when measuring RD cost */47
48
for (int cgScanPos = cgLastScanPos + 1; cgScanPos < (int)cgNum ; cgScanPos++)49
{50
X265_CHECK(coeffNum[cgScanPos] == 0, "count of coeff failure\n");51
-52
uint32_t scanPosBase = (cgScanPos << MLS_CG_SIZE);53
uint32_t blkPos = codeParams.scan[scanPosBase];54
-55
- // TODO: we can't SIMD optimize because PSYVALUE need 64-bits multiplication, convert to Double can work faster by FMA56
- for (int y = 0; y < MLS_CG_SIZE; y++)57
+ bool enable512 = detect512();58
+ if (enable512)59
+ primitives.cu[log2TrSize - 2].psyRdoQuant(m_resiDctCoeff, m_fencDctCoeff, costUncoded, &totalUncodedCost, &totalRdCost, &psyScale, blkPos);60
+ else61
{62
- for (int x = 0; x < MLS_CG_SIZE; x++)63
- {64
- int signCoef = m_resiDctCoeff[blkPos + x]; /* pre-quantization DCT coeff */65
- int predictedCoef = m_fencDctCoeff[blkPos + x] - signCoef; /* predicted DCT = source DCT - residual DCT*/66
-67
- costUncoded[blkPos + x] = ((int64_t)signCoef * signCoef) << scaleBits;68
-69
- /* when no residual coefficient is coded, predicted coef == recon coef */70
- costUncoded[blkPos + x] -= PSYVALUE(predictedCoef);71
-72
- totalUncodedCost += costUncoded[blkPos + x];73
- totalRdCost += costUncoded[blkPos + x];74
- }75
- blkPos += trSize;76
+ primitives.cu[log2TrSize - 2].psyRdoQuant_1p(m_resiDctCoeff, costUncoded, &totalUncodedCost, &totalRdCost,blkPos);77
+ primitives.cu[log2TrSize - 2].psyRdoQuant_2p(m_resiDctCoeff, m_fencDctCoeff, costUncoded, &totalUncodedCost, &totalRdCost, &psyScale, blkPos);78
}79
}80
}81
82
for (int cgScanPos = cgLastScanPos + 1; cgScanPos < (int)cgNum ; cgScanPos++)83
{84
X265_CHECK(coeffNum[cgScanPos] == 0, "count of coeff failure\n");85
-86
uint32_t scanPosBase = (cgScanPos << MLS_CG_SIZE);87
uint32_t blkPos = codeParams.scan[scanPosBase];88
-89
- for (int y = 0; y < MLS_CG_SIZE; y++)90
- {91
- for (int x = 0; x < MLS_CG_SIZE; x++)92
- {93
- int signCoef = m_resiDctCoeff[blkPos + x]; /* pre-quantization DCT coeff */94
- costUncoded[blkPos + x] = ((int64_t)signCoef * signCoef) << scaleBits;95
-96
- totalUncodedCost += costUncoded[blkPos + x];97
- totalRdCost += costUncoded[blkPos + x];98
- }99
- blkPos += trSize;100
- }101
+ primitives.cu[log2TrSize - 2].nonPsyRdoQuant(m_resiDctCoeff, costUncoded, &totalUncodedCost, &totalRdCost, blkPos);102
}103
}104
-105
static const uint8_t table_cnt[5][SCAN_SET_SIZE] =106
{107
// patternSigCtx = 0108
109
// TODO: does we need zero-coeff cost?110
const uint32_t scanPosBase = (cgScanPos << MLS_CG_SIZE);111
uint32_t blkPos = codeParams.scan[scanPosBase];112
-113
if (usePsyMask)114
{115
- // TODO: we can't SIMD optimize because PSYVALUE need 64-bits multiplication, convert to Double can work faster by FMA116
+ bool enable512 = detect512();117
+118
+ if (enable512)119
+ primitives.cu[log2TrSize - 2].psyRdoQuant(m_resiDctCoeff, m_fencDctCoeff, costUncoded, &totalUncodedCost, &totalRdCost, &psyScale, blkPos);120
+ else121
+ {122
+ primitives.cu[log2TrSize - 2].psyRdoQuant_1p(m_resiDctCoeff, costUncoded, &totalUncodedCost, &totalRdCost, blkPos);123
+ primitives.cu[log2TrSize - 2].psyRdoQuant_2p(m_resiDctCoeff, m_fencDctCoeff, costUncoded, &totalUncodedCost, &totalRdCost, &psyScale, blkPos);124
+ }125
+ blkPos = codeParams.scan[scanPosBase];126
for (int y = 0; y < MLS_CG_SIZE; y++)127
{128
for (int x = 0; x < MLS_CG_SIZE; x++)129
{130
- int signCoef = m_resiDctCoeff[blkPos + x]; /* pre-quantization DCT coeff */131
- int predictedCoef = m_fencDctCoeff[blkPos + x] - signCoef; /* predicted DCT = source DCT - residual DCT*/132
-133
- costUncoded[blkPos + x] = ((int64_t)signCoef * signCoef) << scaleBits;134
-135
- /* when no residual coefficient is coded, predicted coef == recon coef */136
- costUncoded[blkPos + x] -= PSYVALUE(predictedCoef);137
-138
- totalUncodedCost += costUncoded[blkPos + x];139
- totalRdCost += costUncoded[blkPos + x];140
-141
const uint32_t scanPosOffset = y * MLS_CG_SIZE + x;142
const uint32_t ctxSig = table_cnt[patternSigCtx][g_scan4x4[codeParams.scanType][scanPosOffset]] + ctxSigOffset;143
X265_CHECK(trSize > 4, "trSize check failure\n");144
145
else146
{147
// non-psy path148
+ primitives.cu[log2TrSize - 2].nonPsyRdoQuant(m_resiDctCoeff, costUncoded, &totalUncodedCost, &totalRdCost, blkPos);149
+ blkPos = codeParams.scan[scanPosBase];150
for (int y = 0; y < MLS_CG_SIZE; y++)151
{152
for (int x = 0; x < MLS_CG_SIZE; x++)153
{154
- int signCoef = m_resiDctCoeff[blkPos + x]; /* pre-quantization DCT coeff */155
- costUncoded[blkPos + x] = ((int64_t)signCoef * signCoef) << scaleBits;156
-157
- totalUncodedCost += costUncoded[blkPos + x];158
- totalRdCost += costUncoded[blkPos + x];159
-160
const uint32_t scanPosOffset = y * MLS_CG_SIZE + x;161
const uint32_t ctxSig = table_cnt[patternSigCtx][g_scan4x4[codeParams.scanType][scanPosOffset]] + ctxSigOffset;162
X265_CHECK(trSize > 4, "trSize check failure\n");163
x265_2.7.tar.gz/source/common/slice.cpp -> x265_2.9.tar.gz/source/common/slice.cpp
Changed
10
1
2
for (int yuv = 0; yuv < 3; yuv++)3
{4
WeightParam& wp = m_weightPredTable[l][i][yuv];5
- wp.bPresentFlag = false;6
+ wp.wtPresent = 0;7
wp.log2WeightDenom = 0;8
wp.inputWeight = 1;9
wp.inputOffset = 0;10
x265_2.7.tar.gz/source/common/slice.h -> x265_2.9.tar.gz/source/common/slice.h
Changed
37
1
2
uint32_t log2WeightDenom;3
int inputWeight;4
int inputOffset;5
- bool bPresentFlag;6
+ int wtPresent;7
8
/* makes a non-h265 weight (i.e. fix7), into an h265 weight */9
void setFromWeightAndOffset(int w, int o, int denom, bool bNormalize)10
11
(w).inputWeight = (s); \12
(w).log2WeightDenom = (d); \13
(w).inputOffset = (o); \14
- (w).bPresentFlag = (b); \15
+ (w).wtPresent = (b); \16
}17
18
class Slice19
20
bool getRapPicFlag() const21
{22
return m_nalUnitType == NAL_UNIT_CODED_SLICE_IDR_W_RADL23
+ || m_nalUnitType == NAL_UNIT_CODED_SLICE_IDR_N_LP24
|| m_nalUnitType == NAL_UNIT_CODED_SLICE_CRA;25
}26
-27
bool getIdrPicFlag() const28
{29
- return m_nalUnitType == NAL_UNIT_CODED_SLICE_IDR_W_RADL;30
+ return m_nalUnitType == NAL_UNIT_CODED_SLICE_IDR_W_RADL31
+ || m_nalUnitType == NAL_UNIT_CODED_SLICE_IDR_N_LP;32
}33
-34
bool isIRAP() const { return m_nalUnitType >= 16 && m_nalUnitType <= 23; }35
36
bool isIntra() const { return m_sliceType == I_SLICE; }37
x265_2.7.tar.gz/source/common/x86/asm-primitives.cpp -> x265_2.9.tar.gz/source/common/x86/asm-primitives.cpp
Changed
2936
1
2
p.chroma[X265_CSP_I422].cu[BLOCK_422_8x16].sa8d = PFX(pixel_sa8d_8x16_ ## cpu); \3
p.chroma[X265_CSP_I422].cu[BLOCK_422_16x32].sa8d = PFX(pixel_sa8d_16x32_ ## cpu); \4
p.chroma[X265_CSP_I422].cu[BLOCK_422_32x64].sa8d = PFX(pixel_sa8d_32x64_ ## cpu)5
-6
#define PIXEL_AVG(cpu) \7
- p.pu[LUMA_64x64].pixelavg_pp = PFX(pixel_avg_64x64_ ## cpu); \8
- p.pu[LUMA_64x48].pixelavg_pp = PFX(pixel_avg_64x48_ ## cpu); \9
- p.pu[LUMA_64x32].pixelavg_pp = PFX(pixel_avg_64x32_ ## cpu); \10
- p.pu[LUMA_64x16].pixelavg_pp = PFX(pixel_avg_64x16_ ## cpu); \11
- p.pu[LUMA_48x64].pixelavg_pp = PFX(pixel_avg_48x64_ ## cpu); \12
- p.pu[LUMA_32x64].pixelavg_pp = PFX(pixel_avg_32x64_ ## cpu); \13
- p.pu[LUMA_32x32].pixelavg_pp = PFX(pixel_avg_32x32_ ## cpu); \14
- p.pu[LUMA_32x24].pixelavg_pp = PFX(pixel_avg_32x24_ ## cpu); \15
- p.pu[LUMA_32x16].pixelavg_pp = PFX(pixel_avg_32x16_ ## cpu); \16
- p.pu[LUMA_32x8].pixelavg_pp = PFX(pixel_avg_32x8_ ## cpu); \17
- p.pu[LUMA_24x32].pixelavg_pp = PFX(pixel_avg_24x32_ ## cpu); \18
- p.pu[LUMA_16x64].pixelavg_pp = PFX(pixel_avg_16x64_ ## cpu); \19
- p.pu[LUMA_16x32].pixelavg_pp = PFX(pixel_avg_16x32_ ## cpu); \20
- p.pu[LUMA_16x16].pixelavg_pp = PFX(pixel_avg_16x16_ ## cpu); \21
- p.pu[LUMA_16x12].pixelavg_pp = PFX(pixel_avg_16x12_ ## cpu); \22
- p.pu[LUMA_16x8].pixelavg_pp = PFX(pixel_avg_16x8_ ## cpu); \23
- p.pu[LUMA_16x4].pixelavg_pp = PFX(pixel_avg_16x4_ ## cpu); \24
- p.pu[LUMA_12x16].pixelavg_pp = PFX(pixel_avg_12x16_ ## cpu); \25
- p.pu[LUMA_8x32].pixelavg_pp = PFX(pixel_avg_8x32_ ## cpu); \26
- p.pu[LUMA_8x16].pixelavg_pp = PFX(pixel_avg_8x16_ ## cpu); \27
- p.pu[LUMA_8x8].pixelavg_pp = PFX(pixel_avg_8x8_ ## cpu); \28
- p.pu[LUMA_8x4].pixelavg_pp = PFX(pixel_avg_8x4_ ## cpu);29
-30
+ p.pu[LUMA_64x64].pixelavg_pp[NONALIGNED] = PFX(pixel_avg_64x64_ ## cpu); \31
+ p.pu[LUMA_64x48].pixelavg_pp[NONALIGNED] = PFX(pixel_avg_64x48_ ## cpu); \32
+ p.pu[LUMA_64x32].pixelavg_pp[NONALIGNED] = PFX(pixel_avg_64x32_ ## cpu); \33
+ p.pu[LUMA_64x16].pixelavg_pp[NONALIGNED] = PFX(pixel_avg_64x16_ ## cpu); \34
+ p.pu[LUMA_48x64].pixelavg_pp[NONALIGNED] = PFX(pixel_avg_48x64_ ## cpu); \35
+ p.pu[LUMA_32x64].pixelavg_pp[NONALIGNED] = PFX(pixel_avg_32x64_ ## cpu); \36
+ p.pu[LUMA_32x32].pixelavg_pp[NONALIGNED] = PFX(pixel_avg_32x32_ ## cpu); \37
+ p.pu[LUMA_32x24].pixelavg_pp[NONALIGNED] = PFX(pixel_avg_32x24_ ## cpu); \38
+ p.pu[LUMA_32x16].pixelavg_pp[NONALIGNED] = PFX(pixel_avg_32x16_ ## cpu); \39
+ p.pu[LUMA_32x8].pixelavg_pp[NONALIGNED] = PFX(pixel_avg_32x8_ ## cpu); \40
+ p.pu[LUMA_24x32].pixelavg_pp[NONALIGNED] = PFX(pixel_avg_24x32_ ## cpu); \41
+ p.pu[LUMA_16x64].pixelavg_pp[NONALIGNED] = PFX(pixel_avg_16x64_ ## cpu); \42
+ p.pu[LUMA_16x32].pixelavg_pp[NONALIGNED] = PFX(pixel_avg_16x32_ ## cpu); \43
+ p.pu[LUMA_16x16].pixelavg_pp[NONALIGNED] = PFX(pixel_avg_16x16_ ## cpu); \44
+ p.pu[LUMA_16x12].pixelavg_pp[NONALIGNED] = PFX(pixel_avg_16x12_ ## cpu); \45
+ p.pu[LUMA_16x8].pixelavg_pp[NONALIGNED] = PFX(pixel_avg_16x8_ ## cpu); \46
+ p.pu[LUMA_16x4].pixelavg_pp[NONALIGNED] = PFX(pixel_avg_16x4_ ## cpu); \47
+ p.pu[LUMA_12x16].pixelavg_pp[NONALIGNED] = PFX(pixel_avg_12x16_ ## cpu); \48
+ p.pu[LUMA_8x32].pixelavg_pp[NONALIGNED] = PFX(pixel_avg_8x32_ ## cpu); \49
+ p.pu[LUMA_8x16].pixelavg_pp[NONALIGNED] = PFX(pixel_avg_8x16_ ## cpu); \50
+ p.pu[LUMA_8x8].pixelavg_pp[NONALIGNED] = PFX(pixel_avg_8x8_ ## cpu); \51
+ p.pu[LUMA_8x4].pixelavg_pp[NONALIGNED] = PFX(pixel_avg_8x4_ ## cpu); \52
+ p.pu[LUMA_64x64].pixelavg_pp[ALIGNED] = PFX(pixel_avg_64x64_ ## cpu); \53
+ p.pu[LUMA_64x48].pixelavg_pp[ALIGNED] = PFX(pixel_avg_64x48_ ## cpu); \54
+ p.pu[LUMA_64x32].pixelavg_pp[ALIGNED] = PFX(pixel_avg_64x32_ ## cpu); \55
+ p.pu[LUMA_64x16].pixelavg_pp[ALIGNED] = PFX(pixel_avg_64x16_ ## cpu); \56
+ p.pu[LUMA_48x64].pixelavg_pp[ALIGNED] = PFX(pixel_avg_48x64_ ## cpu); \57
+ p.pu[LUMA_32x64].pixelavg_pp[ALIGNED] = PFX(pixel_avg_32x64_ ## cpu); \58
+ p.pu[LUMA_32x32].pixelavg_pp[ALIGNED] = PFX(pixel_avg_32x32_ ## cpu); \59
+ p.pu[LUMA_32x24].pixelavg_pp[ALIGNED] = PFX(pixel_avg_32x24_ ## cpu); \60
+ p.pu[LUMA_32x16].pixelavg_pp[ALIGNED] = PFX(pixel_avg_32x16_ ## cpu); \61
+ p.pu[LUMA_32x8].pixelavg_pp[ALIGNED] = PFX(pixel_avg_32x8_ ## cpu); \62
+ p.pu[LUMA_24x32].pixelavg_pp[ALIGNED] = PFX(pixel_avg_24x32_ ## cpu); \63
+ p.pu[LUMA_16x64].pixelavg_pp[ALIGNED] = PFX(pixel_avg_16x64_ ## cpu); \64
+ p.pu[LUMA_16x32].pixelavg_pp[ALIGNED] = PFX(pixel_avg_16x32_ ## cpu); \65
+ p.pu[LUMA_16x16].pixelavg_pp[ALIGNED] = PFX(pixel_avg_16x16_ ## cpu); \66
+ p.pu[LUMA_16x12].pixelavg_pp[ALIGNED] = PFX(pixel_avg_16x12_ ## cpu); \67
+ p.pu[LUMA_16x8].pixelavg_pp[ALIGNED] = PFX(pixel_avg_16x8_ ## cpu); \68
+ p.pu[LUMA_16x4].pixelavg_pp[ALIGNED] = PFX(pixel_avg_16x4_ ## cpu); \69
+ p.pu[LUMA_12x16].pixelavg_pp[ALIGNED] = PFX(pixel_avg_12x16_ ## cpu); \70
+ p.pu[LUMA_8x32].pixelavg_pp[ALIGNED] = PFX(pixel_avg_8x32_ ## cpu); \71
+ p.pu[LUMA_8x16].pixelavg_pp[ALIGNED] = PFX(pixel_avg_8x16_ ## cpu); \72
+ p.pu[LUMA_8x8].pixelavg_pp[ALIGNED] = PFX(pixel_avg_8x8_ ## cpu); \73
+ p.pu[LUMA_8x4].pixelavg_pp[ALIGNED] = PFX(pixel_avg_8x4_ ## cpu);74
#define PIXEL_AVG_W4(cpu) \75
- p.pu[LUMA_4x4].pixelavg_pp = PFX(pixel_avg_4x4_ ## cpu); \76
- p.pu[LUMA_4x8].pixelavg_pp = PFX(pixel_avg_4x8_ ## cpu); \77
- p.pu[LUMA_4x16].pixelavg_pp = PFX(pixel_avg_4x16_ ## cpu);78
-79
+ p.pu[LUMA_4x4].pixelavg_pp[NONALIGNED] = PFX(pixel_avg_4x4_ ## cpu); \80
+ p.pu[LUMA_4x8].pixelavg_pp[NONALIGNED] = PFX(pixel_avg_4x8_ ## cpu); \81
+ p.pu[LUMA_4x16].pixelavg_pp[NONALIGNED] = PFX(pixel_avg_4x16_ ## cpu); \82
+ p.pu[LUMA_4x4].pixelavg_pp[ALIGNED] = PFX(pixel_avg_4x4_ ## cpu); \83
+ p.pu[LUMA_4x8].pixelavg_pp[ALIGNED] = PFX(pixel_avg_4x8_ ## cpu); \84
+ p.pu[LUMA_4x16].pixelavg_pp[ALIGNED] = PFX(pixel_avg_4x16_ ## cpu);85
#define CHROMA_420_FILTERS(cpu) \86
ALL_CHROMA_420_PU(filter_hpp, interp_4tap_horiz_pp, cpu); \87
ALL_CHROMA_420_PU(filter_hps, interp_4tap_horiz_ps, cpu); \88
89
90
#define LUMA_PIXELSUB(cpu) \91
p.cu[BLOCK_4x4].sub_ps = PFX(pixel_sub_ps_4x4_ ## cpu); \92
- p.cu[BLOCK_4x4].add_ps = PFX(pixel_add_ps_4x4_ ## cpu); \93
+ p.cu[BLOCK_4x4].add_ps[NONALIGNED] = PFX(pixel_add_ps_4x4_ ## cpu); \94
+ p.cu[BLOCK_4x4].add_ps[ALIGNED] = PFX(pixel_add_ps_4x4_ ## cpu); \95
ALL_LUMA_CU(sub_ps, pixel_sub_ps, cpu); \96
- ALL_LUMA_CU(add_ps, pixel_add_ps, cpu);97
+ ALL_LUMA_CU(add_ps[NONALIGNED], pixel_add_ps, cpu); \98
+ ALL_LUMA_CU(add_ps[ALIGNED], pixel_add_ps, cpu);99
100
#define CHROMA_420_PIXELSUB_PS(cpu) \101
ALL_CHROMA_420_CU(sub_ps, pixel_sub_ps, cpu); \102
- ALL_CHROMA_420_CU(add_ps, pixel_add_ps, cpu);103
+ ALL_CHROMA_420_CU(add_ps[NONALIGNED], pixel_add_ps, cpu); \104
+ ALL_CHROMA_420_CU(add_ps[ALIGNED], pixel_add_ps, cpu);105
106
#define CHROMA_422_PIXELSUB_PS(cpu) \107
ALL_CHROMA_422_CU(sub_ps, pixel_sub_ps, cpu); \108
- ALL_CHROMA_422_CU(add_ps, pixel_add_ps, cpu);109
+ ALL_CHROMA_422_CU(add_ps[NONALIGNED], pixel_add_ps, cpu); \110
+ ALL_CHROMA_422_CU(add_ps[ALIGNED], pixel_add_ps, cpu);111
112
#define LUMA_VAR(cpu) ALL_LUMA_CU(var, pixel_var, cpu)113
114
-#define LUMA_ADDAVG(cpu) ALL_LUMA_PU(addAvg, addAvg, cpu); p.pu[LUMA_4x4].addAvg = PFX(addAvg_4x4_ ## cpu)115
-#define CHROMA_420_ADDAVG(cpu) ALL_CHROMA_420_PU(addAvg, addAvg, cpu);116
-#define CHROMA_422_ADDAVG(cpu) ALL_CHROMA_422_PU(addAvg, addAvg, cpu);117
+#define LUMA_ADDAVG(cpu) ALL_LUMA_PU(addAvg[NONALIGNED], addAvg, cpu); \118
+ p.pu[LUMA_4x4].addAvg[NONALIGNED] = PFX(addAvg_4x4_ ## cpu); \119
+ ALL_LUMA_PU(addAvg[ALIGNED], addAvg, cpu); \120
+ p.pu[LUMA_4x4].addAvg[ALIGNED] = PFX(addAvg_4x4_ ## cpu)121
+#define CHROMA_420_ADDAVG(cpu) ALL_CHROMA_420_PU(addAvg[NONALIGNED], addAvg, cpu); \122
+ ALL_CHROMA_420_PU(addAvg[ALIGNED], addAvg, cpu)123
+#define CHROMA_422_ADDAVG(cpu) ALL_CHROMA_422_PU(addAvg[NONALIGNED], addAvg, cpu); \124
+ ALL_CHROMA_422_PU(addAvg[ALIGNED], addAvg, cpu)125
126
#define SETUP_INTRA_ANG_COMMON(mode, fno, cpu) \127
p.cu[BLOCK_4x4].intra_pred[mode] = PFX(intra_pred_ang4_ ## fno ## _ ## cpu); \128
129
ALL_CHROMA_444_PU(filter_hpp, interp_4tap_horiz_pp, cpu); \130
ALL_CHROMA_444_PU(filter_hps, interp_4tap_horiz_ps, cpu);131
132
+#define ASSIGN2(func, fname) \133
+ func[ALIGNED] = PFX(fname); \134
+ func[NONALIGNED] = PFX(fname)135
+136
namespace X265_NS {137
// private x265 namespace138
139
140
141
void setupAssemblyPrimitives(EncoderPrimitives &p, int cpuMask) // Main10142
{143
-#if !defined(X86_64)144
-#error "Unsupported build configuration (32bit x86 and HIGH_BIT_DEPTH), you must configure ENABLE_ASSEMBLY=OFF"145
-#endif146
-147
#if X86_64148
p.scanPosLast = PFX(scanPosLast_x64);149
#endif150
151
CHROMA_422_VERT_FILTERS(_sse2);152
CHROMA_444_VERT_FILTERS(sse2);153
154
+#if X86_64155
ALL_LUMA_PU(luma_hpp, interp_8tap_horiz_pp, sse2);156
p.pu[LUMA_4x4].luma_hpp = PFX(interp_8tap_horiz_pp_4x4_sse2);157
ALL_LUMA_PU(luma_hps, interp_8tap_horiz_ps, sse2);158
p.pu[LUMA_4x4].luma_hps = PFX(interp_8tap_horiz_ps_4x4_sse2);159
ALL_LUMA_PU(luma_vpp, interp_8tap_vert_pp, sse2);160
ALL_LUMA_PU(luma_vps, interp_8tap_vert_ps, sse2);161
+#endif162
163
p.ssim_4x4x2_core = PFX(pixel_ssim_4x4x2_core_sse2);164
p.ssim_end_4 = PFX(pixel_ssim_end4_sse2);165
- PIXEL_AVG(sse2);166
+ ASSIGN2(p.pu[LUMA_64x64].pixelavg_pp, pixel_avg_64x64_sse2);167
+ ASSIGN2(p.pu[LUMA_64x48].pixelavg_pp, pixel_avg_64x48_sse2);168
+ ASSIGN2(p.pu[LUMA_64x32].pixelavg_pp, pixel_avg_64x32_sse2);169
+ ASSIGN2(p.pu[LUMA_64x16].pixelavg_pp, pixel_avg_64x16_sse2);170
+ ASSIGN2(p.pu[LUMA_48x64].pixelavg_pp, pixel_avg_48x64_sse2);171
+ ASSIGN2(p.pu[LUMA_32x64].pixelavg_pp, pixel_avg_32x64_sse2);172
+ ASSIGN2(p.pu[LUMA_32x32].pixelavg_pp, pixel_avg_32x32_sse2);173
+ ASSIGN2(p.pu[LUMA_32x24].pixelavg_pp, pixel_avg_32x24_sse2);174
+ ASSIGN2(p.pu[LUMA_32x16].pixelavg_pp, pixel_avg_32x16_sse2);175
+ ASSIGN2(p.pu[LUMA_32x8].pixelavg_pp, pixel_avg_32x8_sse2);176
+ ASSIGN2(p.pu[LUMA_24x32].pixelavg_pp, pixel_avg_24x32_sse2);177
+ ASSIGN2(p.pu[LUMA_16x64].pixelavg_pp, pixel_avg_16x64_sse2);178
+ ASSIGN2(p.pu[LUMA_16x32].pixelavg_pp, pixel_avg_16x32_sse2);179
+ ASSIGN2(p.pu[LUMA_16x16].pixelavg_pp, pixel_avg_16x16_sse2);180
+ ASSIGN2(p.pu[LUMA_16x12].pixelavg_pp, pixel_avg_16x12_sse2);181
+ ASSIGN2(p.pu[LUMA_16x8].pixelavg_pp, pixel_avg_16x8_sse2);182
+ ASSIGN2(p.pu[LUMA_16x4].pixelavg_pp, pixel_avg_16x4_sse2);183
+ ASSIGN2(p.pu[LUMA_12x16].pixelavg_pp, pixel_avg_12x16_sse2);184
+#if X86_64185
+ ASSIGN2(p.pu[LUMA_8x32].pixelavg_pp, pixel_avg_8x32_sse2);186
+ ASSIGN2(p.pu[LUMA_8x16].pixelavg_pp, pixel_avg_8x16_sse2);187
+ ASSIGN2(p.pu[LUMA_8x8].pixelavg_pp, pixel_avg_8x8_sse2);188
+ ASSIGN2(p.pu[LUMA_8x4].pixelavg_pp, pixel_avg_8x4_sse2);189
+#endif190
PIXEL_AVG_W4(mmx2);191
LUMA_VAR(sse2);192
193
194
- ALL_LUMA_TU(blockfill_s, blockfill_s, sse2);195
+ ALL_LUMA_TU(blockfill_s[ALIGNED], blockfill_s, sse2);196
+ ALL_LUMA_TU(blockfill_s[NONALIGNED], blockfill_s, sse2);197
ALL_LUMA_TU_S(cpy1Dto2D_shr, cpy1Dto2D_shr_, sse2);198
- ALL_LUMA_TU_S(cpy1Dto2D_shl, cpy1Dto2D_shl_, sse2);199
+ ALL_LUMA_TU_S(cpy1Dto2D_shl[ALIGNED], cpy1Dto2D_shl_, sse2);200
+ ALL_LUMA_TU_S(cpy1Dto2D_shl[NONALIGNED], cpy1Dto2D_shl_, sse2);201
ALL_LUMA_TU_S(cpy2Dto1D_shr, cpy2Dto1D_shr_, sse2);202
ALL_LUMA_TU_S(cpy2Dto1D_shl, cpy2Dto1D_shl_, sse2);203
- ALL_LUMA_TU_S(ssd_s, pixel_ssd_s_, sse2);204
- ALL_LUMA_TU_S(calcresidual, getResidual, sse2);205
+#if X86_64206
+ ASSIGN2(p.cu[BLOCK_4x4].ssd_s,pixel_ssd_s_4_sse2 );207
+ ASSIGN2(p.cu[BLOCK_8x8].ssd_s,pixel_ssd_s_8_sse2);208
+ ASSIGN2(p.cu[BLOCK_16x16].ssd_s,pixel_ssd_s_16_sse2);209
+ ASSIGN2(p.cu[BLOCK_32x32].ssd_s,pixel_ssd_s_32_sse2 );210
+#endif211
+ ALL_LUMA_TU_S(calcresidual[ALIGNED], getResidual, sse2);212
+ ALL_LUMA_TU_S(calcresidual[NONALIGNED], getResidual, sse2);213
ALL_LUMA_TU_S(transpose, transpose, sse2);214
215
p.cu[BLOCK_4x4].intra_pred[PLANAR_IDX] = PFX(intra_pred_planar4_sse2);216
p.cu[BLOCK_8x8].intra_pred[PLANAR_IDX] = PFX(intra_pred_planar8_sse2);217
p.cu[BLOCK_16x16].intra_pred[PLANAR_IDX] = PFX(intra_pred_planar16_sse2);218
+#if X86_64219
p.cu[BLOCK_32x32].intra_pred[PLANAR_IDX] = PFX(intra_pred_planar32_sse2);220
ALL_LUMA_TU_S(intra_pred[DC_IDX], intra_pred_dc, sse2);221
-222
+#endif223
p.cu[BLOCK_4x4].intra_pred[2] = PFX(intra_pred_ang4_2_sse2);224
p.cu[BLOCK_4x4].intra_pred[3] = PFX(intra_pred_ang4_3_sse2);225
p.cu[BLOCK_4x4].intra_pred[4] = PFX(intra_pred_ang4_4_sse2);226
227
p.cu[BLOCK_4x4].intra_pred[23] = PFX(intra_pred_ang4_23_sse2);228
p.cu[BLOCK_4x4].intra_pred[24] = PFX(intra_pred_ang4_24_sse2);229
p.cu[BLOCK_4x4].intra_pred[25] = PFX(intra_pred_ang4_25_sse2);230
+#if X86_64231
p.cu[BLOCK_4x4].intra_pred[26] = PFX(intra_pred_ang4_26_sse2);232
+#endif233
p.cu[BLOCK_4x4].intra_pred[27] = PFX(intra_pred_ang4_27_sse2);234
p.cu[BLOCK_4x4].intra_pred[28] = PFX(intra_pred_ang4_28_sse2);235
p.cu[BLOCK_4x4].intra_pred[29] = PFX(intra_pred_ang4_29_sse2);236
237
p.cu[BLOCK_4x4].intra_pred[32] = PFX(intra_pred_ang4_32_sse2);238
p.cu[BLOCK_4x4].intra_pred[33] = PFX(intra_pred_ang4_33_sse2);239
240
+#if X86_64 && X265_DEPTH <= 10241
+ p.cu[BLOCK_4x4].sse_ss = PFX(pixel_ssd_ss_4x4_mmx2);242
p.chroma[X265_CSP_I422].cu[BLOCK_422_32x64].sse_pp = (pixel_sse_t)PFX(pixel_ssd_ss_32x64_sse2);243
p.chroma[X265_CSP_I422].cu[BLOCK_422_4x8].sse_pp = (pixel_sse_t)PFX(pixel_ssd_ss_4x8_mmx2);244
p.chroma[X265_CSP_I422].cu[BLOCK_422_8x16].sse_pp = (pixel_sse_t)PFX(pixel_ssd_ss_8x16_sse2);245
p.chroma[X265_CSP_I422].cu[BLOCK_422_16x32].sse_pp = (pixel_sse_t)PFX(pixel_ssd_ss_16x32_sse2);246
-#if X265_DEPTH <= 10247
- p.cu[BLOCK_4x4].sse_ss = PFX(pixel_ssd_ss_4x4_mmx2);248
- ALL_LUMA_CU(sse_ss, pixel_ssd_ss, sse2);249
+250
+ p.cu[BLOCK_8x8].sse_ss = PFX(pixel_ssd_ss_8x8_sse2);251
+ p.cu[BLOCK_16x16].sse_ss = PFX(pixel_ssd_ss_16x16_sse2);252
+ p.cu[BLOCK_32x32].sse_ss = PFX(pixel_ssd_ss_32x32_sse2);253
+ p.cu[BLOCK_64x64].sse_ss = PFX(pixel_ssd_ss_64x64_sse2);254
#endif255
p.cu[BLOCK_4x4].dct = PFX(dct4_sse2);256
p.cu[BLOCK_8x8].dct = PFX(dct8_sse2);257
p.cu[BLOCK_4x4].idct = PFX(idct4_sse2);258
+#if X86_64259
p.cu[BLOCK_8x8].idct = PFX(idct8_sse2);260
-261
+#endif262
p.idst4x4 = PFX(idst4_sse2);263
p.dst4x4 = PFX(dst4_sse2);264
265
266
//p.planecopy_sp = PFX(downShift_16_sse2);267
p.planecopy_sp_shl = PFX(upShift_16_sse2);268
269
- ALL_CHROMA_420_PU(p2s, filterPixelToShort, sse2);270
- ALL_CHROMA_422_PU(p2s, filterPixelToShort, sse2);271
- ALL_CHROMA_444_PU(p2s, filterPixelToShort, sse2);272
- ALL_LUMA_PU(convert_p2s, filterPixelToShort, sse2);273
+ ALL_CHROMA_420_PU(p2s[ALIGNED], filterPixelToShort, sse2);274
+ ALL_CHROMA_422_PU(p2s[ALIGNED], filterPixelToShort, sse2);275
+ ALL_CHROMA_444_PU(p2s[ALIGNED], filterPixelToShort, sse2);276
+ ALL_LUMA_PU(convert_p2s[ALIGNED], filterPixelToShort, sse2);277
+ ALL_CHROMA_420_PU(p2s[NONALIGNED], filterPixelToShort, sse2);278
+ ALL_CHROMA_422_PU(p2s[NONALIGNED], filterPixelToShort, sse2);279
+ ALL_CHROMA_444_PU(p2s[NONALIGNED], filterPixelToShort, sse2);280
+ ALL_LUMA_PU(convert_p2s[NONALIGNED], filterPixelToShort, sse2);281
ALL_LUMA_TU(count_nonzero, count_nonzero, sse2);282
p.propagateCost = PFX(mbtree_propagate_cost_sse2);283
}284
if (cpuMask & X265_CPU_SSE3)285
{286
+#if X86_64287
ALL_CHROMA_420_PU(filter_hpp, interp_4tap_horiz_pp, sse3);288
ALL_CHROMA_422_PU(filter_hpp, interp_4tap_horiz_pp, sse3);289
ALL_CHROMA_444_PU(filter_hpp, interp_4tap_horiz_pp, sse3);290
ALL_CHROMA_420_PU(filter_hps, interp_4tap_horiz_ps, sse3);291
ALL_CHROMA_422_PU(filter_hps, interp_4tap_horiz_ps, sse3);292
ALL_CHROMA_444_PU(filter_hps, interp_4tap_horiz_ps, sse3);293
+#endif294
}295
if (cpuMask & X265_CPU_SSSE3)296
{297
- p.scale1D_128to64 = PFX(scale1D_128to64_ssse3);298
+ ASSIGN2(p.scale1D_128to64, scale1D_128to64_ssse3);299
p.scale2D_64to32 = PFX(scale2D_64to32_ssse3);300
301
// p.pu[LUMA_4x4].satd = p.cu[BLOCK_4x4].sa8d = PFX(pixel_satd_4x4_ssse3); this one is broken302
303
304
p.frameInitLowres = PFX(frame_init_lowres_core_ssse3);305
306
- ALL_LUMA_PU(convert_p2s, filterPixelToShort, ssse3);307
+ ALL_LUMA_PU(convert_p2s[ALIGNED], filterPixelToShort, ssse3);308
+ ALL_LUMA_PU(convert_p2s[NONALIGNED], filterPixelToShort, ssse3);309
+310
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_4x4].p2s, filterPixelToShort_4x4_ssse3);311
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_4x8].p2s, filterPixelToShort_4x8_ssse3);312
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_4x16].p2s, filterPixelToShort_4x16_ssse3);313
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_8x4].p2s, filterPixelToShort_8x4_ssse3);314
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_8x8].p2s, filterPixelToShort_8x8_ssse3);315
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_8x16].p2s, filterPixelToShort_8x16_ssse3);316
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_8x32].p2s, filterPixelToShort_8x32_ssse3);317
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_16x4].p2s, filterPixelToShort_16x4_ssse3);318
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_16x8].p2s, filterPixelToShort_16x8_ssse3);319
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_16x12].p2s, filterPixelToShort_16x12_ssse3);320
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_16x16].p2s, filterPixelToShort_16x16_ssse3);321
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_16x32].p2s, filterPixelToShort_16x32_ssse3);322
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_32x8].p2s, filterPixelToShort_32x8_ssse3);323
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_32x16].p2s, filterPixelToShort_32x16_ssse3);324
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_32x24].p2s, filterPixelToShort_32x24_ssse3);325
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_32x32].p2s, filterPixelToShort_32x32_ssse3);326
+327
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_4x4].p2s, filterPixelToShort_4x4_ssse3);328
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_4x8].p2s, filterPixelToShort_4x8_ssse3);329
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_4x16].p2s, filterPixelToShort_4x16_ssse3);330
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_4x32].p2s, filterPixelToShort_4x32_ssse3);331
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_8x4].p2s, filterPixelToShort_8x4_ssse3);332
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_8x8].p2s, filterPixelToShort_8x8_ssse3);333
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_8x12].p2s, filterPixelToShort_8x12_ssse3);334
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_8x16].p2s, filterPixelToShort_8x16_ssse3);335
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_8x32].p2s, filterPixelToShort_8x32_ssse3);336
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_8x64].p2s, filterPixelToShort_8x64_ssse3);337
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_12x32].p2s, filterPixelToShort_12x32_ssse3);338
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_16x8].p2s, filterPixelToShort_16x8_ssse3);339
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_16x16].p2s, filterPixelToShort_16x16_ssse3);340
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_16x24].p2s, filterPixelToShort_16x24_ssse3);341
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_16x32].p2s, filterPixelToShort_16x32_ssse3);342
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_16x64].p2s, filterPixelToShort_16x64_ssse3);343
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_24x64].p2s, filterPixelToShort_24x64_ssse3);344
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].p2s, filterPixelToShort_32x16_ssse3);345
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_32x32].p2s, filterPixelToShort_32x32_ssse3);346
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_32x48].p2s, filterPixelToShort_32x48_ssse3);347
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_32x64].p2s, filterPixelToShort_32x64_ssse3);348
+349
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_4x2].p2s, filterPixelToShort_4x2_ssse3);350
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_8x2].p2s, filterPixelToShort_8x2_ssse3);351
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_8x6].p2s, filterPixelToShort_8x6_ssse3);352
353
- p.chroma[X265_CSP_I420].pu[CHROMA_420_4x4].p2s = PFX(filterPixelToShort_4x4_ssse3);354
- p.chroma[X265_CSP_I420].pu[CHROMA_420_4x8].p2s = PFX(filterPixelToShort_4x8_ssse3);355
- p.chroma[X265_CSP_I420].pu[CHROMA_420_4x16].p2s = PFX(filterPixelToShort_4x16_ssse3);356
- p.chroma[X265_CSP_I420].pu[CHROMA_420_8x4].p2s = PFX(filterPixelToShort_8x4_ssse3);357
- p.chroma[X265_CSP_I420].pu[CHROMA_420_8x8].p2s = PFX(filterPixelToShort_8x8_ssse3);358
- p.chroma[X265_CSP_I420].pu[CHROMA_420_8x16].p2s = PFX(filterPixelToShort_8x16_ssse3);359
- p.chroma[X265_CSP_I420].pu[CHROMA_420_8x32].p2s = PFX(filterPixelToShort_8x32_ssse3);360
- p.chroma[X265_CSP_I420].pu[CHROMA_420_16x4].p2s = PFX(filterPixelToShort_16x4_ssse3);361
- p.chroma[X265_CSP_I420].pu[CHROMA_420_16x8].p2s = PFX(filterPixelToShort_16x8_ssse3);362
- p.chroma[X265_CSP_I420].pu[CHROMA_420_16x12].p2s = PFX(filterPixelToShort_16x12_ssse3);363
- p.chroma[X265_CSP_I420].pu[CHROMA_420_16x16].p2s = PFX(filterPixelToShort_16x16_ssse3);364
- p.chroma[X265_CSP_I420].pu[CHROMA_420_16x32].p2s = PFX(filterPixelToShort_16x32_ssse3);365
- p.chroma[X265_CSP_I420].pu[CHROMA_420_32x8].p2s = PFX(filterPixelToShort_32x8_ssse3);366
- p.chroma[X265_CSP_I420].pu[CHROMA_420_32x16].p2s = PFX(filterPixelToShort_32x16_ssse3);367
- p.chroma[X265_CSP_I420].pu[CHROMA_420_32x24].p2s = PFX(filterPixelToShort_32x24_ssse3);368
- p.chroma[X265_CSP_I420].pu[CHROMA_420_32x32].p2s = PFX(filterPixelToShort_32x32_ssse3);369
- p.chroma[X265_CSP_I422].pu[CHROMA_422_4x4].p2s = PFX(filterPixelToShort_4x4_ssse3);370
- p.chroma[X265_CSP_I422].pu[CHROMA_422_4x8].p2s = PFX(filterPixelToShort_4x8_ssse3);371
- p.chroma[X265_CSP_I422].pu[CHROMA_422_4x16].p2s = PFX(filterPixelToShort_4x16_ssse3);372
- p.chroma[X265_CSP_I422].pu[CHROMA_422_4x32].p2s = PFX(filterPixelToShort_4x32_ssse3);373
- p.chroma[X265_CSP_I422].pu[CHROMA_422_8x4].p2s = PFX(filterPixelToShort_8x4_ssse3);374
- p.chroma[X265_CSP_I422].pu[CHROMA_422_8x8].p2s = PFX(filterPixelToShort_8x8_ssse3);375
- p.chroma[X265_CSP_I422].pu[CHROMA_422_8x12].p2s = PFX(filterPixelToShort_8x12_ssse3);376
- p.chroma[X265_CSP_I422].pu[CHROMA_422_8x16].p2s = PFX(filterPixelToShort_8x16_ssse3);377
- p.chroma[X265_CSP_I422].pu[CHROMA_422_8x32].p2s = PFX(filterPixelToShort_8x32_ssse3);378
- p.chroma[X265_CSP_I422].pu[CHROMA_422_8x64].p2s = PFX(filterPixelToShort_8x64_ssse3);379
- p.chroma[X265_CSP_I422].pu[CHROMA_422_12x32].p2s = PFX(filterPixelToShort_12x32_ssse3);380
- p.chroma[X265_CSP_I422].pu[CHROMA_422_16x8].p2s = PFX(filterPixelToShort_16x8_ssse3);381
- p.chroma[X265_CSP_I422].pu[CHROMA_422_16x16].p2s = PFX(filterPixelToShort_16x16_ssse3);382
- p.chroma[X265_CSP_I422].pu[CHROMA_422_16x24].p2s = PFX(filterPixelToShort_16x24_ssse3);383
- p.chroma[X265_CSP_I422].pu[CHROMA_422_16x32].p2s = PFX(filterPixelToShort_16x32_ssse3);384
- p.chroma[X265_CSP_I422].pu[CHROMA_422_16x64].p2s = PFX(filterPixelToShort_16x64_ssse3);385
- p.chroma[X265_CSP_I422].pu[CHROMA_422_24x64].p2s = PFX(filterPixelToShort_24x64_ssse3);386
- p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].p2s = PFX(filterPixelToShort_32x16_ssse3);387
- p.chroma[X265_CSP_I422].pu[CHROMA_422_32x32].p2s = PFX(filterPixelToShort_32x32_ssse3);388
- p.chroma[X265_CSP_I422].pu[CHROMA_422_32x48].p2s = PFX(filterPixelToShort_32x48_ssse3);389
- p.chroma[X265_CSP_I422].pu[CHROMA_422_32x64].p2s = PFX(filterPixelToShort_32x64_ssse3);390
- p.chroma[X265_CSP_I420].pu[CHROMA_420_4x2].p2s = PFX(filterPixelToShort_4x2_ssse3);391
- p.chroma[X265_CSP_I420].pu[CHROMA_420_8x2].p2s = PFX(filterPixelToShort_8x2_ssse3);392
- p.chroma[X265_CSP_I420].pu[CHROMA_420_8x6].p2s = PFX(filterPixelToShort_8x6_ssse3);393
p.findPosFirstLast = PFX(findPosFirstLast_ssse3);394
p.fix8Unpack = PFX(cutree_fix8_unpack_ssse3);395
p.fix8Pack = PFX(cutree_fix8_pack_ssse3);396
}397
if (cpuMask & X265_CPU_SSE4)398
{399
+#if X86_64400
p.pelFilterLumaStrong[0] = PFX(pelFilterLumaStrong_V_sse4);401
p.pelFilterLumaStrong[1] = PFX(pelFilterLumaStrong_H_sse4);402
p.pelFilterChroma[0] = PFX(pelFilterChroma_V_sse4);403
p.pelFilterChroma[1] = PFX(pelFilterChroma_H_sse4);404
-405
p.saoCuOrgE0 = PFX(saoCuOrgE0_sse4);406
+#endif407
p.saoCuOrgE1 = PFX(saoCuOrgE1_sse4);408
p.saoCuOrgE1_2Rows = PFX(saoCuOrgE1_2Rows_sse4);409
p.saoCuOrgE2[0] = PFX(saoCuOrgE2_sse4);410
411
CHROMA_422_ADDAVG(sse4);412
413
LUMA_FILTERS(sse4);414
+415
+#if X86_64416
+ p.pu[LUMA_4x4].luma_hpp = PFX(interp_8tap_horiz_pp_4x4_sse4);417
+ p.pu[LUMA_4x8].luma_hpp = PFX(interp_8tap_horiz_pp_4x8_sse4);418
+ p.pu[LUMA_4x16].luma_hpp = PFX(interp_8tap_horiz_pp_4x16_sse4);419
+ p.pu[LUMA_4x4].luma_hps = PFX(interp_8tap_horiz_ps_4x4_sse4);420
+ p.pu[LUMA_4x8].luma_hps = PFX(interp_8tap_horiz_ps_4x8_sse4);421
+ p.pu[LUMA_4x16].luma_hps = PFX(interp_8tap_horiz_ps_4x16_sse4);422
+#endif423
+424
+ p.pu[LUMA_8x8].luma_hpp = PFX(interp_8tap_horiz_pp_8x8_sse4);425
+ p.pu[LUMA_16x16].luma_hpp = PFX(interp_8tap_horiz_pp_16x16_sse4);426
+ p.pu[LUMA_32x32].luma_hpp = PFX(interp_8tap_horiz_pp_32x32_sse4);427
+ p.pu[LUMA_64x64].luma_hpp = PFX(interp_8tap_horiz_pp_64x64_sse4);428
+ p.pu[LUMA_8x4].luma_hpp = PFX(interp_8tap_horiz_pp_8x4_sse4);429
+430
+ p.pu[LUMA_16x8].luma_hpp = PFX(interp_8tap_horiz_pp_16x8_sse4);431
+ p.pu[LUMA_8x16].luma_hpp = PFX(interp_8tap_horiz_pp_8x16_sse4);432
+ p.pu[LUMA_16x32].luma_hpp = PFX(interp_8tap_horiz_pp_16x32_sse4);433
+ p.pu[LUMA_32x16].luma_hpp = PFX(interp_8tap_horiz_pp_32x16_sse4);434
+ p.pu[LUMA_64x32].luma_hpp = PFX(interp_8tap_horiz_pp_64x32_sse4);435
+ p.pu[LUMA_32x64].luma_hpp = PFX(interp_8tap_horiz_pp_32x64_sse4);436
+ p.pu[LUMA_16x12].luma_hpp = PFX(interp_8tap_horiz_pp_16x12_sse4);437
+ p.pu[LUMA_12x16].luma_hpp = PFX(interp_8tap_horiz_pp_12x16_sse4);438
+ p.pu[LUMA_16x4].luma_hpp = PFX(interp_8tap_horiz_pp_16x4_sse4);439
+440
+ p.pu[LUMA_32x24].luma_hpp = PFX(interp_8tap_horiz_pp_32x24_sse4);441
+ p.pu[LUMA_24x32].luma_hpp = PFX(interp_8tap_horiz_pp_24x32_sse4);442
+ p.pu[LUMA_32x8].luma_hpp = PFX(interp_8tap_horiz_pp_32x8_sse4);443
+ p.pu[LUMA_8x32].luma_hpp = PFX(interp_8tap_horiz_pp_8x32_sse4);444
+ p.pu[LUMA_64x48].luma_hpp = PFX(interp_8tap_horiz_pp_64x48_sse4);445
+ p.pu[LUMA_48x64].luma_hpp = PFX(interp_8tap_horiz_pp_48x64_sse4);446
+ p.pu[LUMA_64x16].luma_hpp = PFX(interp_8tap_horiz_pp_64x16_sse4);447
+ p.pu[LUMA_16x64].luma_hpp = PFX(interp_8tap_horiz_pp_16x64_sse4);448
+449
+ p.pu[LUMA_8x8].luma_hps = PFX(interp_8tap_horiz_ps_8x8_sse4);450
+ p.pu[LUMA_16x16].luma_hps = PFX(interp_8tap_horiz_ps_16x16_sse4);451
+ p.pu[LUMA_32x32].luma_hps = PFX(interp_8tap_horiz_ps_32x32_sse4);452
+ p.pu[LUMA_64x64].luma_hps = PFX(interp_8tap_horiz_ps_64x64_sse4);453
+ p.pu[LUMA_8x4].luma_hps = PFX(interp_8tap_horiz_ps_8x4_sse4);454
+ p.pu[LUMA_16x8].luma_hps = PFX(interp_8tap_horiz_ps_16x8_sse4);455
+ p.pu[LUMA_8x16].luma_hps = PFX(interp_8tap_horiz_ps_8x16_sse4);456
+ p.pu[LUMA_16x32].luma_hps = PFX(interp_8tap_horiz_ps_16x32_sse4);457
+ p.pu[LUMA_32x16].luma_hps = PFX(interp_8tap_horiz_ps_32x16_sse4);458
+ p.pu[LUMA_64x32].luma_hps = PFX(interp_8tap_horiz_ps_64x32_sse4);459
+ p.pu[LUMA_32x64].luma_hps = PFX(interp_8tap_horiz_ps_32x64_sse4);460
+ p.pu[LUMA_16x12].luma_hps = PFX(interp_8tap_horiz_ps_16x12_sse4);461
+ p.pu[LUMA_12x16].luma_hps = PFX(interp_8tap_horiz_ps_12x16_sse4);462
+ p.pu[LUMA_16x4].luma_hps = PFX(interp_8tap_horiz_ps_16x4_sse4);463
+ p.pu[LUMA_32x24].luma_hps = PFX(interp_8tap_horiz_ps_32x24_sse4);464
+ p.pu[LUMA_24x32].luma_hps = PFX(interp_8tap_horiz_ps_24x32_sse4);465
+ p.pu[LUMA_32x8].luma_hps = PFX(interp_8tap_horiz_ps_32x8_sse4);466
+ p.pu[LUMA_8x32].luma_hps = PFX(interp_8tap_horiz_ps_8x32_sse4);467
+ p.pu[LUMA_64x48].luma_hps = PFX(interp_8tap_horiz_ps_64x48_sse4);468
+ p.pu[LUMA_48x64].luma_hps = PFX(interp_8tap_horiz_ps_48x64_sse4);469
+ p.pu[LUMA_64x16].luma_hps = PFX(interp_8tap_horiz_ps_64x16_sse4);470
+ p.pu[LUMA_16x64].luma_hps = PFX(interp_8tap_horiz_ps_16x64_sse4);471
+472
+ ALL_LUMA_PU(luma_vpp, interp_8tap_vert_pp, sse4); p.pu[LUMA_4x4].luma_vpp = PFX(interp_8tap_vert_pp_4x4_sse4);473
+ ALL_LUMA_PU(luma_vps, interp_8tap_vert_ps, sse4); p.pu[LUMA_4x4].luma_vps = PFX(interp_8tap_vert_ps_4x4_sse4);474
+ ALL_LUMA_PU(luma_vsp, interp_8tap_vert_sp, sse4); p.pu[LUMA_4x4].luma_vsp = PFX(interp_8tap_vert_sp_4x4_sse4);475
+ ALL_LUMA_PU_T(luma_hvpp, interp_8tap_hv_pp_cpu); p.pu[LUMA_4x4].luma_hvpp = interp_8tap_hv_pp_cpu<LUMA_4x4>;476
CHROMA_420_HORIZ_FILTERS(sse4);477
CHROMA_420_VERT_FILTERS_SSE4(_sse4);478
CHROMA_422_HORIZ_FILTERS(_sse4);479
480
481
// TODO: check POPCNT flag!482
ALL_LUMA_TU_S(copy_cnt, copy_cnt_, sse4);483
-#if X265_DEPTH <= 10484
+#if X86_64 && X265_DEPTH <= 10485
ALL_LUMA_CU(psy_cost_pp, psyCost_pp, sse4);486
#endif487
488
- p.chroma[X265_CSP_I420].pu[CHROMA_420_2x4].p2s = PFX(filterPixelToShort_2x4_sse4);489
- p.chroma[X265_CSP_I420].pu[CHROMA_420_2x8].p2s = PFX(filterPixelToShort_2x8_sse4);490
- p.chroma[X265_CSP_I420].pu[CHROMA_420_6x8].p2s = PFX(filterPixelToShort_6x8_sse4);491
- p.chroma[X265_CSP_I422].pu[CHROMA_422_2x8].p2s = PFX(filterPixelToShort_2x8_sse4);492
- p.chroma[X265_CSP_I422].pu[CHROMA_422_2x16].p2s = PFX(filterPixelToShort_2x16_sse4);493
- p.chroma[X265_CSP_I422].pu[CHROMA_422_6x16].p2s = PFX(filterPixelToShort_6x16_sse4);494
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_2x4].p2s[NONALIGNED] = PFX(filterPixelToShort_2x4_sse4);495
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_2x8].p2s[NONALIGNED] = PFX(filterPixelToShort_2x8_sse4);496
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_6x8].p2s[NONALIGNED] = PFX(filterPixelToShort_6x8_sse4);497
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_2x8].p2s[NONALIGNED] = PFX(filterPixelToShort_2x8_sse4);498
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_2x16].p2s[NONALIGNED] = PFX(filterPixelToShort_2x16_sse4);499
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_6x16].p2s[NONALIGNED] = PFX(filterPixelToShort_6x16_sse4);500
p.costCoeffRemain = PFX(costCoeffRemain_sse4);501
#if X86_64502
p.saoCuStatsE0 = PFX(saoCuStatsE0_sse4);503
504
p.saoCuStatsE3 = PFX(saoCuStatsE3_sse4);505
#endif506
}507
+#if X86_64508
if (cpuMask & X265_CPU_AVX)509
{510
// p.pu[LUMA_4x4].satd = p.cu[BLOCK_4x4].sa8d = PFX(pixel_satd_4x4_avx); fails tests511
512
p.cu[BLOCK_32x32].intra_pred[32] = PFX(intra_pred_ang32_32_avx2);513
p.cu[BLOCK_32x32].intra_pred[33] = PFX(intra_pred_ang32_33_avx2);514
p.cu[BLOCK_32x32].intra_pred[34] = PFX(intra_pred_ang32_2_avx2);515
-516
- p.pu[LUMA_12x16].pixelavg_pp = PFX(pixel_avg_12x16_avx2);517
- p.pu[LUMA_16x4].pixelavg_pp = PFX(pixel_avg_16x4_avx2);518
- p.pu[LUMA_16x8].pixelavg_pp = PFX(pixel_avg_16x8_avx2);519
- p.pu[LUMA_16x12].pixelavg_pp = PFX(pixel_avg_16x12_avx2);520
- p.pu[LUMA_16x16].pixelavg_pp = PFX(pixel_avg_16x16_avx2);521
- p.pu[LUMA_16x32].pixelavg_pp = PFX(pixel_avg_16x32_avx2);522
- p.pu[LUMA_16x64].pixelavg_pp = PFX(pixel_avg_16x64_avx2);523
- p.pu[LUMA_24x32].pixelavg_pp = PFX(pixel_avg_24x32_avx2);524
- p.pu[LUMA_32x8].pixelavg_pp = PFX(pixel_avg_32x8_avx2);525
- p.pu[LUMA_32x16].pixelavg_pp = PFX(pixel_avg_32x16_avx2);526
- p.pu[LUMA_32x24].pixelavg_pp = PFX(pixel_avg_32x24_avx2);527
- p.pu[LUMA_32x32].pixelavg_pp = PFX(pixel_avg_32x32_avx2);528
- p.pu[LUMA_32x64].pixelavg_pp = PFX(pixel_avg_32x64_avx2);529
- p.pu[LUMA_64x16].pixelavg_pp = PFX(pixel_avg_64x16_avx2);530
- p.pu[LUMA_64x32].pixelavg_pp = PFX(pixel_avg_64x32_avx2);531
- p.pu[LUMA_64x48].pixelavg_pp = PFX(pixel_avg_64x48_avx2);532
- p.pu[LUMA_64x64].pixelavg_pp = PFX(pixel_avg_64x64_avx2);533
- p.pu[LUMA_48x64].pixelavg_pp = PFX(pixel_avg_48x64_avx2);534
-535
- p.pu[LUMA_8x4].addAvg = PFX(addAvg_8x4_avx2);536
- p.pu[LUMA_8x8].addAvg = PFX(addAvg_8x8_avx2);537
- p.pu[LUMA_8x16].addAvg = PFX(addAvg_8x16_avx2);538
- p.pu[LUMA_8x32].addAvg = PFX(addAvg_8x32_avx2);539
- p.pu[LUMA_12x16].addAvg = PFX(addAvg_12x16_avx2);540
- p.pu[LUMA_16x4].addAvg = PFX(addAvg_16x4_avx2);541
- p.pu[LUMA_16x8].addAvg = PFX(addAvg_16x8_avx2);542
- p.pu[LUMA_16x12].addAvg = PFX(addAvg_16x12_avx2);543
- p.pu[LUMA_16x16].addAvg = PFX(addAvg_16x16_avx2);544
- p.pu[LUMA_16x32].addAvg = PFX(addAvg_16x32_avx2);545
- p.pu[LUMA_16x64].addAvg = PFX(addAvg_16x64_avx2);546
- p.pu[LUMA_24x32].addAvg = PFX(addAvg_24x32_avx2);547
- p.pu[LUMA_32x8].addAvg = PFX(addAvg_32x8_avx2);548
- p.pu[LUMA_32x16].addAvg = PFX(addAvg_32x16_avx2);549
- p.pu[LUMA_32x24].addAvg = PFX(addAvg_32x24_avx2);550
- p.pu[LUMA_32x32].addAvg = PFX(addAvg_32x32_avx2);551
- p.pu[LUMA_32x64].addAvg = PFX(addAvg_32x64_avx2);552
- p.pu[LUMA_48x64].addAvg = PFX(addAvg_48x64_avx2);553
- p.pu[LUMA_64x16].addAvg = PFX(addAvg_64x16_avx2);554
- p.pu[LUMA_64x32].addAvg = PFX(addAvg_64x32_avx2);555
- p.pu[LUMA_64x48].addAvg = PFX(addAvg_64x48_avx2);556
- p.pu[LUMA_64x64].addAvg = PFX(addAvg_64x64_avx2);557
-558
- p.chroma[X265_CSP_I420].pu[CHROMA_420_8x2].addAvg = PFX(addAvg_8x2_avx2);559
- p.chroma[X265_CSP_I420].pu[CHROMA_420_8x4].addAvg = PFX(addAvg_8x4_avx2);560
- p.chroma[X265_CSP_I420].pu[CHROMA_420_8x6].addAvg = PFX(addAvg_8x6_avx2);561
- p.chroma[X265_CSP_I420].pu[CHROMA_420_8x8].addAvg = PFX(addAvg_8x8_avx2);562
- p.chroma[X265_CSP_I420].pu[CHROMA_420_8x16].addAvg = PFX(addAvg_8x16_avx2);563
- p.chroma[X265_CSP_I420].pu[CHROMA_420_8x32].addAvg = PFX(addAvg_8x32_avx2);564
- p.chroma[X265_CSP_I420].pu[CHROMA_420_12x16].addAvg = PFX(addAvg_12x16_avx2);565
- p.chroma[X265_CSP_I420].pu[CHROMA_420_16x4].addAvg = PFX(addAvg_16x4_avx2);566
- p.chroma[X265_CSP_I420].pu[CHROMA_420_16x8].addAvg = PFX(addAvg_16x8_avx2);567
- p.chroma[X265_CSP_I420].pu[CHROMA_420_16x12].addAvg = PFX(addAvg_16x12_avx2);568
- p.chroma[X265_CSP_I420].pu[CHROMA_420_16x16].addAvg = PFX(addAvg_16x16_avx2);569
- p.chroma[X265_CSP_I420].pu[CHROMA_420_16x32].addAvg = PFX(addAvg_16x32_avx2);570
- p.chroma[X265_CSP_I420].pu[CHROMA_420_32x8].addAvg = PFX(addAvg_32x8_avx2);571
- p.chroma[X265_CSP_I420].pu[CHROMA_420_32x16].addAvg = PFX(addAvg_32x16_avx2);572
- p.chroma[X265_CSP_I420].pu[CHROMA_420_32x24].addAvg = PFX(addAvg_32x24_avx2);573
- p.chroma[X265_CSP_I420].pu[CHROMA_420_32x32].addAvg = PFX(addAvg_32x32_avx2);574
-575
- p.chroma[X265_CSP_I422].pu[CHROMA_422_8x16].addAvg = PFX(addAvg_8x16_avx2);576
- p.chroma[X265_CSP_I422].pu[CHROMA_422_16x32].addAvg = PFX(addAvg_16x32_avx2);577
- p.chroma[X265_CSP_I422].pu[CHROMA_422_32x64].addAvg = PFX(addAvg_32x64_avx2);578
- p.chroma[X265_CSP_I422].pu[CHROMA_422_8x8].addAvg = PFX(addAvg_8x8_avx2);579
- p.chroma[X265_CSP_I422].pu[CHROMA_422_16x16].addAvg = PFX(addAvg_16x16_avx2);580
- p.chroma[X265_CSP_I422].pu[CHROMA_422_8x32].addAvg = PFX(addAvg_8x32_avx2);581
- p.chroma[X265_CSP_I422].pu[CHROMA_422_32x32].addAvg = PFX(addAvg_32x32_avx2);582
- p.chroma[X265_CSP_I422].pu[CHROMA_422_16x64].addAvg = PFX(addAvg_16x64_avx2);583
- p.chroma[X265_CSP_I422].pu[CHROMA_422_8x12].addAvg = PFX(addAvg_8x12_avx2);584
- p.chroma[X265_CSP_I422].pu[CHROMA_422_8x4].addAvg = PFX(addAvg_8x4_avx2);585
- p.chroma[X265_CSP_I422].pu[CHROMA_422_16x24].addAvg = PFX(addAvg_16x24_avx2);586
- p.chroma[X265_CSP_I422].pu[CHROMA_422_16x8].addAvg = PFX(addAvg_16x8_avx2);587
- p.chroma[X265_CSP_I422].pu[CHROMA_422_8x64].addAvg = PFX(addAvg_8x64_avx2);588
- p.chroma[X265_CSP_I422].pu[CHROMA_422_24x64].addAvg = PFX(addAvg_24x64_avx2);589
- p.chroma[X265_CSP_I422].pu[CHROMA_422_12x32].addAvg = PFX(addAvg_12x32_avx2);590
- p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].addAvg = PFX(addAvg_32x16_avx2);591
- p.chroma[X265_CSP_I422].pu[CHROMA_422_32x48].addAvg = PFX(addAvg_32x48_avx2);592
+ ASSIGN2(p.pu[LUMA_12x16].pixelavg_pp, pixel_avg_12x16_avx2);593
+ ASSIGN2(p.pu[LUMA_16x4].pixelavg_pp, pixel_avg_16x4_avx2);594
+ ASSIGN2(p.pu[LUMA_16x8].pixelavg_pp, pixel_avg_16x8_avx2);595
+ ASSIGN2(p.pu[LUMA_16x12].pixelavg_pp, pixel_avg_16x12_avx2);596
+ ASSIGN2(p.pu[LUMA_16x16].pixelavg_pp, pixel_avg_16x16_avx2);597
+ ASSIGN2(p.pu[LUMA_16x32].pixelavg_pp, pixel_avg_16x32_avx2);598
+ ASSIGN2(p.pu[LUMA_16x64].pixelavg_pp, pixel_avg_16x64_avx2);599
+ ASSIGN2(p.pu[LUMA_24x32].pixelavg_pp, pixel_avg_24x32_avx2);600
+ ASSIGN2(p.pu[LUMA_32x8].pixelavg_pp, pixel_avg_32x8_avx2);601
+ ASSIGN2(p.pu[LUMA_32x16].pixelavg_pp, pixel_avg_32x16_avx2);602
+ ASSIGN2(p.pu[LUMA_32x24].pixelavg_pp, pixel_avg_32x24_avx2);603
+ ASSIGN2(p.pu[LUMA_32x32].pixelavg_pp, pixel_avg_32x32_avx2);604
+ ASSIGN2(p.pu[LUMA_32x64].pixelavg_pp, pixel_avg_32x64_avx2);605
+ ASSIGN2(p.pu[LUMA_64x16].pixelavg_pp, pixel_avg_64x16_avx2);606
+ ASSIGN2(p.pu[LUMA_64x32].pixelavg_pp, pixel_avg_64x32_avx2);607
+ ASSIGN2(p.pu[LUMA_64x48].pixelavg_pp, pixel_avg_64x48_avx2);608
+ ASSIGN2(p.pu[LUMA_64x64].pixelavg_pp, pixel_avg_64x64_avx2);609
+ ASSIGN2(p.pu[LUMA_48x64].pixelavg_pp, pixel_avg_48x64_avx2);610
+ ASSIGN2(p.pu[LUMA_8x4].addAvg, addAvg_8x4_avx2);611
+ ASSIGN2(p.pu[LUMA_8x8].addAvg, addAvg_8x8_avx2);612
+ ASSIGN2(p.pu[LUMA_8x16].addAvg, addAvg_8x16_avx2);613
+ ASSIGN2(p.pu[LUMA_8x32].addAvg, addAvg_8x32_avx2);614
+ ASSIGN2(p.pu[LUMA_12x16].addAvg, addAvg_12x16_avx2);615
+ ASSIGN2(p.pu[LUMA_16x4].addAvg, addAvg_16x4_avx2);616
+ ASSIGN2(p.pu[LUMA_16x8].addAvg, addAvg_16x8_avx2);617
+ ASSIGN2(p.pu[LUMA_16x12].addAvg, addAvg_16x12_avx2);618
+ ASSIGN2(p.pu[LUMA_16x16].addAvg, addAvg_16x16_avx2);619
+ ASSIGN2(p.pu[LUMA_16x32].addAvg, addAvg_16x32_avx2);620
+ ASSIGN2(p.pu[LUMA_16x64].addAvg, addAvg_16x64_avx2);621
+ ASSIGN2(p.pu[LUMA_24x32].addAvg, addAvg_24x32_avx2);622
+ ASSIGN2(p.pu[LUMA_32x8].addAvg, addAvg_32x8_avx2);623
+ ASSIGN2(p.pu[LUMA_32x16].addAvg, addAvg_32x16_avx2);624
+ ASSIGN2(p.pu[LUMA_32x24].addAvg, addAvg_32x24_avx2);625
+ ASSIGN2(p.pu[LUMA_32x32].addAvg, addAvg_32x32_avx2);626
+ ASSIGN2(p.pu[LUMA_32x64].addAvg, addAvg_32x64_avx2);627
+ ASSIGN2(p.pu[LUMA_48x64].addAvg, addAvg_48x64_avx2);628
+ ASSIGN2(p.pu[LUMA_64x16].addAvg, addAvg_64x16_avx2);629
+ ASSIGN2(p.pu[LUMA_64x32].addAvg, addAvg_64x32_avx2);630
+ ASSIGN2(p.pu[LUMA_64x48].addAvg, addAvg_64x48_avx2);631
+ ASSIGN2(p.pu[LUMA_64x64].addAvg, addAvg_64x64_avx2);632
+633
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_8x2].addAvg, addAvg_8x2_avx2);634
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_8x4].addAvg, addAvg_8x4_avx2);635
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_8x6].addAvg, addAvg_8x6_avx2);636
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_8x8].addAvg, addAvg_8x8_avx2);637
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_8x16].addAvg, addAvg_8x16_avx2);638
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_8x32].addAvg, addAvg_8x32_avx2);639
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_12x16].addAvg, addAvg_12x16_avx2);640
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_16x4].addAvg, addAvg_16x4_avx2);641
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_16x8].addAvg, addAvg_16x8_avx2);642
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_16x12].addAvg, addAvg_16x12_avx2);643
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_16x16].addAvg, addAvg_16x16_avx2);644
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_16x32].addAvg, addAvg_16x32_avx2);645
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_32x8].addAvg, addAvg_32x8_avx2);646
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_32x16].addAvg, addAvg_32x16_avx2);647
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_32x24].addAvg, addAvg_32x24_avx2);648
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_32x32].addAvg, addAvg_32x32_avx2);649
+650
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_8x16].addAvg, addAvg_8x16_avx2);651
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_16x32].addAvg, addAvg_16x32_avx2);652
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_32x64].addAvg, addAvg_32x64_avx2);653
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_8x8].addAvg, addAvg_8x8_avx2);654
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_16x16].addAvg,addAvg_16x16_avx2);655
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_8x32].addAvg, addAvg_8x32_avx2);656
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_32x32].addAvg, addAvg_32x32_avx2);657
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_16x64].addAvg, addAvg_16x64_avx2);658
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_8x12].addAvg, addAvg_8x12_avx2);659
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_8x4].addAvg, addAvg_8x4_avx2);660
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_16x24].addAvg, addAvg_16x24_avx2);661
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_16x8].addAvg, addAvg_16x8_avx2);662
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_8x64].addAvg, addAvg_8x64_avx2);663
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_24x64].addAvg, addAvg_24x64_avx2);664
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_12x32].addAvg, addAvg_12x32_avx2);665
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].addAvg, addAvg_32x16_avx2);666
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_32x48].addAvg, addAvg_32x48_avx2);667
668
p.cu[BLOCK_4x4].psy_cost_pp = PFX(psyCost_pp_4x4_avx2);669
p.cu[BLOCK_16x16].intra_pred[PLANAR_IDX] = PFX(intra_pred_planar16_avx2);670
671
p.chroma[X265_CSP_I422].pu[CHROMA_422_16x8].satd = PFX(pixel_satd_16x8_avx2);672
p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].satd = PFX(pixel_satd_32x16_avx2);673
674
- p.cu[BLOCK_16x16].ssd_s = PFX(pixel_ssd_s_16_avx2);675
- p.cu[BLOCK_32x32].ssd_s = PFX(pixel_ssd_s_32_avx2);676
-677
+ ASSIGN2( p.cu[BLOCK_16x16].ssd_s,pixel_ssd_s_16_avx2);678
+ ASSIGN2( p.cu[BLOCK_32x32].ssd_s,pixel_ssd_s_32_avx2);679
p.cu[BLOCK_16x16].sse_ss = (pixel_sse_ss_t)PFX(pixel_ssd_16x16_avx2);680
p.cu[BLOCK_32x32].sse_ss = (pixel_sse_ss_t)PFX(pixel_ssd_32x32_avx2);681
p.cu[BLOCK_64x64].sse_ss = (pixel_sse_ss_t)PFX(pixel_ssd_64x64_avx2);682
683
p.idst4x4 = PFX(idst4_avx2);684
p.denoiseDct = PFX(denoise_dct_avx2);685
686
- p.scale1D_128to64 = PFX(scale1D_128to64_avx2);687
+ ASSIGN2(p.scale1D_128to64, scale1D_128to64_avx2);688
p.scale2D_64to32 = PFX(scale2D_64to32_avx2);689
690
p.weight_pp = PFX(weight_pp_avx2);691
692
p.sign = PFX(calSign_avx2);693
p.planecopy_cp = PFX(upShift_8_avx2);694
695
- p.cu[BLOCK_16x16].calcresidual = PFX(getResidual16_avx2);696
- p.cu[BLOCK_32x32].calcresidual = PFX(getResidual32_avx2);697
-698
- p.cu[BLOCK_16x16].blockfill_s = PFX(blockfill_s_16x16_avx2);699
- p.cu[BLOCK_32x32].blockfill_s = PFX(blockfill_s_32x32_avx2);700
+ ASSIGN2(p.cu[BLOCK_16x16].calcresidual, getResidual16_avx2);701
+ ASSIGN2(p.cu[BLOCK_32x32].calcresidual, getResidual32_avx2);702
703
+ ASSIGN2(p.cu[BLOCK_16x16].blockfill_s, blockfill_s_16x16_avx2);704
+ ASSIGN2(p.cu[BLOCK_32x32].blockfill_s, blockfill_s_32x32_avx2);705
ALL_LUMA_TU(count_nonzero, count_nonzero, avx2);706
- ALL_LUMA_TU_S(cpy1Dto2D_shl, cpy1Dto2D_shl_, avx2);707
+ ALL_LUMA_TU_S(cpy1Dto2D_shl[ALIGNED], cpy1Dto2D_shl_, avx2);708
+ ALL_LUMA_TU_S(cpy1Dto2D_shl[NONALIGNED], cpy1Dto2D_shl_, avx2);709
ALL_LUMA_TU_S(cpy1Dto2D_shr, cpy1Dto2D_shr_, avx2);710
-711
p.cu[BLOCK_8x8].copy_cnt = PFX(copy_cnt_8_avx2);712
p.cu[BLOCK_16x16].copy_cnt = PFX(copy_cnt_16_avx2);713
p.cu[BLOCK_32x32].copy_cnt = PFX(copy_cnt_32_avx2);714
715
ALL_LUMA_PU(luma_vss, interp_8tap_vert_ss, avx2);716
p.pu[LUMA_4x4].luma_vsp = PFX(interp_8tap_vert_sp_4x4_avx2); // since ALL_LUMA_PU didn't declare 4x4 size, calling separately luma_vsp function to use 717
718
- p.cu[BLOCK_16x16].add_ps = PFX(pixel_add_ps_16x16_avx2);719
- p.cu[BLOCK_32x32].add_ps = PFX(pixel_add_ps_32x32_avx2);720
- p.cu[BLOCK_64x64].add_ps = PFX(pixel_add_ps_64x64_avx2);721
- p.chroma[X265_CSP_I420].cu[BLOCK_420_16x16].add_ps = PFX(pixel_add_ps_16x16_avx2);722
- p.chroma[X265_CSP_I420].cu[BLOCK_420_32x32].add_ps = PFX(pixel_add_ps_32x32_avx2);723
- p.chroma[X265_CSP_I422].cu[BLOCK_422_16x32].add_ps = PFX(pixel_add_ps_16x32_avx2);724
- p.chroma[X265_CSP_I422].cu[BLOCK_422_32x64].add_ps = PFX(pixel_add_ps_32x64_avx2);725
+ ASSIGN2(p.cu[BLOCK_16x16].add_ps, pixel_add_ps_16x16_avx2);726
+ ASSIGN2(p.cu[BLOCK_32x32].add_ps, pixel_add_ps_32x32_avx2);727
+ ASSIGN2(p.cu[BLOCK_64x64].add_ps, pixel_add_ps_64x64_avx2);728
+ ASSIGN2(p.chroma[X265_CSP_I420].cu[BLOCK_420_16x16].add_ps, pixel_add_ps_16x16_avx2);729
+ ASSIGN2(p.chroma[X265_CSP_I420].cu[BLOCK_420_32x32].add_ps, pixel_add_ps_32x32_avx2);730
+ ASSIGN2(p.chroma[X265_CSP_I422].cu[BLOCK_422_16x32].add_ps, pixel_add_ps_16x32_avx2);731
+ ASSIGN2(p.chroma[X265_CSP_I422].cu[BLOCK_422_32x64].add_ps, pixel_add_ps_32x64_avx2);732
733
p.cu[BLOCK_16x16].sub_ps = PFX(pixel_sub_ps_16x16_avx2);734
p.cu[BLOCK_32x32].sub_ps = PFX(pixel_sub_ps_32x32_avx2);735
736
p.pu[LUMA_64x48].sad_x4 = PFX(pixel_sad_x4_64x48_avx2);737
p.pu[LUMA_64x64].sad_x4 = PFX(pixel_sad_x4_64x64_avx2);738
739
- p.pu[LUMA_16x4].convert_p2s = PFX(filterPixelToShort_16x4_avx2);740
- p.pu[LUMA_16x8].convert_p2s = PFX(filterPixelToShort_16x8_avx2);741
- p.pu[LUMA_16x12].convert_p2s = PFX(filterPixelToShort_16x12_avx2);742
- p.pu[LUMA_16x16].convert_p2s = PFX(filterPixelToShort_16x16_avx2);743
- p.pu[LUMA_16x32].convert_p2s = PFX(filterPixelToShort_16x32_avx2);744
- p.pu[LUMA_16x64].convert_p2s = PFX(filterPixelToShort_16x64_avx2);745
- p.pu[LUMA_32x8].convert_p2s = PFX(filterPixelToShort_32x8_avx2);746
- p.pu[LUMA_32x16].convert_p2s = PFX(filterPixelToShort_32x16_avx2);747
- p.pu[LUMA_32x24].convert_p2s = PFX(filterPixelToShort_32x24_avx2);748
- p.pu[LUMA_32x32].convert_p2s = PFX(filterPixelToShort_32x32_avx2);749
- p.pu[LUMA_32x64].convert_p2s = PFX(filterPixelToShort_32x64_avx2);750
- p.pu[LUMA_64x16].convert_p2s = PFX(filterPixelToShort_64x16_avx2);751
- p.pu[LUMA_64x32].convert_p2s = PFX(filterPixelToShort_64x32_avx2);752
- p.pu[LUMA_64x48].convert_p2s = PFX(filterPixelToShort_64x48_avx2);753
- p.pu[LUMA_64x64].convert_p2s = PFX(filterPixelToShort_64x64_avx2);754
- p.pu[LUMA_24x32].convert_p2s = PFX(filterPixelToShort_24x32_avx2);755
- p.pu[LUMA_48x64].convert_p2s = PFX(filterPixelToShort_48x64_avx2);756
-757
- p.chroma[X265_CSP_I420].pu[CHROMA_420_16x4].p2s = PFX(filterPixelToShort_16x4_avx2);758
- p.chroma[X265_CSP_I420].pu[CHROMA_420_16x8].p2s = PFX(filterPixelToShort_16x8_avx2);759
- p.chroma[X265_CSP_I420].pu[CHROMA_420_16x12].p2s = PFX(filterPixelToShort_16x12_avx2);760
- p.chroma[X265_CSP_I420].pu[CHROMA_420_16x16].p2s = PFX(filterPixelToShort_16x16_avx2);761
- p.chroma[X265_CSP_I420].pu[CHROMA_420_16x32].p2s = PFX(filterPixelToShort_16x32_avx2);762
- p.chroma[X265_CSP_I420].pu[CHROMA_420_24x32].p2s = PFX(filterPixelToShort_24x32_avx2);763
- p.chroma[X265_CSP_I420].pu[CHROMA_420_32x8].p2s = PFX(filterPixelToShort_32x8_avx2);764
- p.chroma[X265_CSP_I420].pu[CHROMA_420_32x16].p2s = PFX(filterPixelToShort_32x16_avx2);765
- p.chroma[X265_CSP_I420].pu[CHROMA_420_32x24].p2s = PFX(filterPixelToShort_32x24_avx2);766
- p.chroma[X265_CSP_I420].pu[CHROMA_420_32x32].p2s = PFX(filterPixelToShort_32x32_avx2);767
- p.chroma[X265_CSP_I422].pu[CHROMA_422_16x8].p2s = PFX(filterPixelToShort_16x8_avx2);768
- p.chroma[X265_CSP_I422].pu[CHROMA_422_16x16].p2s = PFX(filterPixelToShort_16x16_avx2);769
- p.chroma[X265_CSP_I422].pu[CHROMA_422_16x24].p2s = PFX(filterPixelToShort_16x24_avx2);770
- p.chroma[X265_CSP_I422].pu[CHROMA_422_16x32].p2s = PFX(filterPixelToShort_16x32_avx2);771
- p.chroma[X265_CSP_I422].pu[CHROMA_422_16x64].p2s = PFX(filterPixelToShort_16x64_avx2);772
- p.chroma[X265_CSP_I422].pu[CHROMA_422_24x64].p2s = PFX(filterPixelToShort_24x64_avx2);773
- p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].p2s = PFX(filterPixelToShort_32x16_avx2);774
- p.chroma[X265_CSP_I422].pu[CHROMA_422_32x32].p2s = PFX(filterPixelToShort_32x32_avx2);775
- p.chroma[X265_CSP_I422].pu[CHROMA_422_32x48].p2s = PFX(filterPixelToShort_32x48_avx2);776
- p.chroma[X265_CSP_I422].pu[CHROMA_422_32x64].p2s = PFX(filterPixelToShort_32x64_avx2);777
+ ASSIGN2(p.pu[LUMA_16x4].convert_p2s, filterPixelToShort_16x4_avx2);778
+ ASSIGN2(p.pu[LUMA_16x8].convert_p2s, filterPixelToShort_16x8_avx2);779
+ ASSIGN2(p.pu[LUMA_16x12].convert_p2s, filterPixelToShort_16x12_avx2);780
+ ASSIGN2(p.pu[LUMA_16x16].convert_p2s, filterPixelToShort_16x16_avx2);781
+ ASSIGN2(p.pu[LUMA_16x32].convert_p2s, filterPixelToShort_16x32_avx2);782
+ ASSIGN2(p.pu[LUMA_16x64].convert_p2s, filterPixelToShort_16x64_avx2);783
+ ASSIGN2(p.pu[LUMA_32x8].convert_p2s, filterPixelToShort_32x8_avx2);784
+ ASSIGN2(p.pu[LUMA_32x16].convert_p2s, filterPixelToShort_32x16_avx2);785
+ ASSIGN2(p.pu[LUMA_32x24].convert_p2s, filterPixelToShort_32x24_avx2);786
+ ASSIGN2(p.pu[LUMA_32x32].convert_p2s, filterPixelToShort_32x32_avx2);787
+ ASSIGN2(p.pu[LUMA_32x64].convert_p2s, filterPixelToShort_32x64_avx2);788
+ ASSIGN2(p.pu[LUMA_64x16].convert_p2s, filterPixelToShort_64x16_avx2);789
+ ASSIGN2(p.pu[LUMA_64x32].convert_p2s, filterPixelToShort_64x32_avx2);790
+ ASSIGN2(p.pu[LUMA_64x48].convert_p2s, filterPixelToShort_64x48_avx2);791
+ ASSIGN2(p.pu[LUMA_64x64].convert_p2s, filterPixelToShort_64x64_avx2);792
+ ASSIGN2(p.pu[LUMA_24x32].convert_p2s, filterPixelToShort_24x32_avx2);793
+ ASSIGN2(p.pu[LUMA_48x64].convert_p2s, filterPixelToShort_48x64_avx2);794
+795
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_16x4].p2s, filterPixelToShort_16x4_avx2);796
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_16x8].p2s, filterPixelToShort_16x8_avx2);797
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_16x12].p2s, filterPixelToShort_16x12_avx2);798
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_16x16].p2s, filterPixelToShort_16x16_avx2);799
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_16x32].p2s, filterPixelToShort_16x32_avx2);800
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_24x32].p2s, filterPixelToShort_24x32_avx2);801
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_32x8].p2s, filterPixelToShort_32x8_avx2);802
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_32x16].p2s, filterPixelToShort_32x16_avx2);803
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_32x24].p2s, filterPixelToShort_32x24_avx2);804
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_32x32].p2s, filterPixelToShort_32x32_avx2);805
+806
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_16x8].p2s, filterPixelToShort_16x8_avx2);807
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_16x16].p2s, filterPixelToShort_16x16_avx2);808
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_16x24].p2s, filterPixelToShort_16x24_avx2);809
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_16x32].p2s, filterPixelToShort_16x32_avx2);810
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_16x64].p2s, filterPixelToShort_16x64_avx2);811
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_24x64].p2s, filterPixelToShort_24x64_avx2);812
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].p2s, filterPixelToShort_32x16_avx2);813
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_32x32].p2s, filterPixelToShort_32x32_avx2);814
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_32x48].p2s, filterPixelToShort_32x48_avx2);815
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_32x64].p2s, filterPixelToShort_32x64_avx2);816
817
p.pu[LUMA_4x4].luma_hps = PFX(interp_8tap_horiz_ps_4x4_avx2);818
p.pu[LUMA_4x8].luma_hps = PFX(interp_8tap_horiz_ps_4x8_avx2);819
820
p.integral_inith[INTEGRAL_8] = PFX(integral8h_avx2);821
p.integral_inith[INTEGRAL_12] = PFX(integral12h_avx2);822
p.integral_inith[INTEGRAL_16] = PFX(integral16h_avx2);823
+ p.cu[BLOCK_4x4].nonPsyRdoQuant = PFX(nonPsyRdoQuant4_avx2);824
+ p.cu[BLOCK_8x8].nonPsyRdoQuant = PFX(nonPsyRdoQuant8_avx2);825
+ p.cu[BLOCK_16x16].nonPsyRdoQuant = PFX(nonPsyRdoQuant16_avx2);826
+ p.cu[BLOCK_32x32].nonPsyRdoQuant = PFX(nonPsyRdoQuant32_avx2);827
+ p.cu[BLOCK_4x4].psyRdoQuant_1p = PFX(psyRdoQuant_1p4_avx2);828
+ p.cu[BLOCK_8x8].psyRdoQuant_1p = PFX(psyRdoQuant_1p8_avx2);829
+ p.cu[BLOCK_16x16].psyRdoQuant_1p = PFX(psyRdoQuant_1p16_avx2);830
+ p.cu[BLOCK_32x32].psyRdoQuant_1p = PFX(psyRdoQuant_1p32_avx2);831
832
/* TODO: This kernel needs to be modified to work with HIGH_BIT_DEPTH only 833
p.planeClipAndMax = PFX(planeClipAndMax_avx2); */834
835
p.costCoeffNxN = PFX(costCoeffNxN_avx2_bmi2);836
}837
}838
+ if (cpuMask & X265_CPU_AVX512)839
+ {840
+ p.cu[BLOCK_16x16].var = PFX(pixel_var_16x16_avx512);841
+ p.cu[BLOCK_32x32].calcresidual[NONALIGNED] = PFX(getResidual32_avx512);842
+ p.cu[BLOCK_32x32].calcresidual[ALIGNED] = PFX(getResidual_aligned32_avx512);843
+ p.cu[BLOCK_64x64].sub_ps = PFX(pixel_sub_ps_64x64_avx512);844
+ p.cu[BLOCK_32x32].sub_ps = PFX(pixel_sub_ps_32x32_avx512);845
+ p.chroma[X265_CSP_I420].cu[BLOCK_420_32x32].sub_ps = PFX(pixel_sub_ps_32x32_avx512);846
+ p.chroma[X265_CSP_I422].cu[BLOCK_422_32x64].sub_ps = PFX(pixel_sub_ps_32x64_avx512);847
+848
+ p.cu[BLOCK_64x64].add_ps[NONALIGNED] = PFX(pixel_add_ps_64x64_avx512);849
+ p.cu[BLOCK_32x32].add_ps[NONALIGNED] = PFX(pixel_add_ps_32x32_avx512);850
+ p.chroma[X265_CSP_I420].cu[BLOCK_420_32x32].add_ps[NONALIGNED] = PFX(pixel_add_ps_32x32_avx512);851
+ p.chroma[X265_CSP_I422].cu[BLOCK_422_32x64].add_ps[NONALIGNED] = PFX(pixel_add_ps_32x64_avx512);852
+853
+ p.cu[BLOCK_32x32].add_ps[ALIGNED] = PFX(pixel_add_ps_aligned_32x32_avx512);854
+ p.cu[BLOCK_64x64].add_ps[ALIGNED] = PFX(pixel_add_ps_aligned_64x64_avx512);855
+ p.chroma[X265_CSP_I420].cu[BLOCK_420_32x32].add_ps[ALIGNED] = PFX(pixel_add_ps_aligned_32x32_avx512);856
+ p.chroma[X265_CSP_I422].cu[BLOCK_422_32x64].add_ps[ALIGNED] = PFX(pixel_add_ps_aligned_32x64_avx512);857
+858
+ // 64 X N859
+ p.cu[BLOCK_64x64].copy_ss = PFX(blockcopy_ss_64x64_avx512);860
+ p.pu[LUMA_64x64].copy_pp = (copy_pp_t)PFX(blockcopy_ss_64x64_avx512);861
+ p.pu[LUMA_64x48].copy_pp = (copy_pp_t)PFX(blockcopy_ss_64x48_avx512);862
+ p.pu[LUMA_64x32].copy_pp = (copy_pp_t)PFX(blockcopy_ss_64x32_avx512);863
+ p.pu[LUMA_64x16].copy_pp = (copy_pp_t)PFX(blockcopy_ss_64x16_avx512);864
+ p.cu[BLOCK_64x64].copy_ps = (copy_ps_t)PFX(blockcopy_ss_64x64_avx512);865
+ p.cu[BLOCK_64x64].copy_sp = (copy_sp_t)PFX(blockcopy_ss_64x64_avx512);866
+867
+ // 32 X N868
+ p.cu[BLOCK_32x32].copy_ss = PFX(blockcopy_ss_32x32_avx512);869
+ p.pu[LUMA_32x64].copy_pp = (copy_pp_t)PFX(blockcopy_ss_32x64_avx512);870
+ p.pu[LUMA_32x32].copy_pp = (copy_pp_t)PFX(blockcopy_ss_32x32_avx512);871
+ p.pu[LUMA_32x24].copy_pp = (copy_pp_t)PFX(blockcopy_ss_32x24_avx512);872
+ p.pu[LUMA_32x16].copy_pp = (copy_pp_t)PFX(blockcopy_ss_32x16_avx512);873
+ p.pu[LUMA_32x8].copy_pp = (copy_pp_t)PFX(blockcopy_ss_32x8_avx512);874
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x8].copy_pp = (copy_pp_t)PFX(blockcopy_ss_32x8_avx512);875
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x16].copy_pp = (copy_pp_t)PFX(blockcopy_ss_32x16_avx512);876
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x24].copy_pp = (copy_pp_t)PFX(blockcopy_ss_32x24_avx512);877
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x32].copy_pp = (copy_pp_t)PFX(blockcopy_ss_32x32_avx512);878
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].copy_pp = (copy_pp_t)PFX(blockcopy_ss_32x16_avx512);879
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x32].copy_pp = (copy_pp_t)PFX(blockcopy_ss_32x32_avx512);880
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x48].copy_pp = (copy_pp_t)PFX(blockcopy_ss_32x48_avx512);881
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x64].copy_pp = (copy_pp_t)PFX(blockcopy_ss_32x64_avx512);882
+ p.chroma[X265_CSP_I420].cu[BLOCK_420_32x32].copy_ss = PFX(blockcopy_ss_32x32_avx512);883
+ p.chroma[X265_CSP_I422].cu[BLOCK_422_32x64].copy_ss = PFX(blockcopy_ss_32x64_avx512);884
+ p.cu[BLOCK_32x32].copy_ps = (copy_ps_t)PFX(blockcopy_ss_32x32_avx512);885
+ p.chroma[X265_CSP_I420].cu[BLOCK_420_32x32].copy_ps = (copy_ps_t)PFX(blockcopy_ss_32x32_avx512);886
+ p.chroma[X265_CSP_I422].cu[BLOCK_422_32x64].copy_ps = (copy_ps_t)PFX(blockcopy_ss_32x64_avx512);887
+ p.cu[BLOCK_32x32].copy_sp = (copy_sp_t)PFX(blockcopy_ss_32x32_avx512);888
+ p.chroma[X265_CSP_I420].cu[BLOCK_420_32x32].copy_sp = (copy_sp_t)PFX(blockcopy_ss_32x32_avx512);889
+ p.chroma[X265_CSP_I422].cu[BLOCK_422_32x64].copy_sp = (copy_sp_t)PFX(blockcopy_ss_32x64_avx512);890
+891
+ p.pu[LUMA_64x16].convert_p2s[NONALIGNED] = PFX(filterPixelToShort_64x16_avx512);892
+ p.pu[LUMA_64x32].convert_p2s[NONALIGNED] = PFX(filterPixelToShort_64x32_avx512);893
+ p.pu[LUMA_64x48].convert_p2s[NONALIGNED] = PFX(filterPixelToShort_64x48_avx512);894
+ p.pu[LUMA_64x64].convert_p2s[NONALIGNED] = PFX(filterPixelToShort_64x64_avx512);895
+ p.pu[LUMA_32x8].convert_p2s[NONALIGNED] = PFX(filterPixelToShort_32x8_avx512);896
+ p.pu[LUMA_32x16].convert_p2s[NONALIGNED] = PFX(filterPixelToShort_32x16_avx512);897
+ p.pu[LUMA_32x24].convert_p2s[NONALIGNED] = PFX(filterPixelToShort_32x24_avx512);898
+ p.pu[LUMA_32x32].convert_p2s[NONALIGNED] = PFX(filterPixelToShort_32x32_avx512);899
+ p.pu[LUMA_32x64].convert_p2s[NONALIGNED] = PFX(filterPixelToShort_32x64_avx512);900
+ p.pu[LUMA_48x64].convert_p2s[NONALIGNED] = PFX(filterPixelToShort_48x64_avx512);901
+902
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_2x4].p2s[ALIGNED] = PFX(filterPixelToShort_2x4_sse4);903
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_2x8].p2s[ALIGNED] = PFX(filterPixelToShort_2x8_sse4);904
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_6x8].p2s[ALIGNED] = PFX(filterPixelToShort_6x8_sse4);905
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x8].p2s[NONALIGNED] = PFX(filterPixelToShort_32x8_avx512);906
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x16].p2s[NONALIGNED] = PFX(filterPixelToShort_32x16_avx512);907
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x24].p2s[NONALIGNED] = PFX(filterPixelToShort_32x24_avx512);908
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x32].p2s[NONALIGNED] = PFX(filterPixelToShort_32x32_avx512);909
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].p2s[NONALIGNED] = PFX(filterPixelToShort_32x16_avx512);910
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x32].p2s[NONALIGNED] = PFX(filterPixelToShort_32x32_avx512);911
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x48].p2s[NONALIGNED] = PFX(filterPixelToShort_32x48_avx512);912
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x64].p2s[NONALIGNED] = PFX(filterPixelToShort_32x64_avx512);913
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_2x8].p2s[ALIGNED] = PFX(filterPixelToShort_2x8_sse4);914
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_2x16].p2s[ALIGNED] = PFX(filterPixelToShort_2x16_sse4);915
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_6x16].p2s[ALIGNED] = PFX(filterPixelToShort_6x16_sse4);916
+ p.chroma[X265_CSP_I444].pu[LUMA_32x8].p2s[NONALIGNED] = PFX(filterPixelToShort_32x8_avx512);917
+ p.chroma[X265_CSP_I444].pu[LUMA_32x16].p2s[NONALIGNED] = PFX(filterPixelToShort_32x16_avx512);918
+ p.chroma[X265_CSP_I444].pu[LUMA_32x24].p2s[NONALIGNED] = PFX(filterPixelToShort_32x24_avx512);919
+ p.chroma[X265_CSP_I444].pu[LUMA_32x32].p2s[NONALIGNED] = PFX(filterPixelToShort_32x32_avx512);920
+ p.chroma[X265_CSP_I444].pu[LUMA_32x64].p2s[NONALIGNED] = PFX(filterPixelToShort_32x64_avx512);921
+ p.chroma[X265_CSP_I444].pu[LUMA_64x16].p2s[NONALIGNED] = PFX(filterPixelToShort_64x16_avx512);922
+ p.chroma[X265_CSP_I444].pu[LUMA_64x32].p2s[NONALIGNED] = PFX(filterPixelToShort_64x32_avx512);923
+ p.chroma[X265_CSP_I444].pu[LUMA_64x48].p2s[NONALIGNED] = PFX(filterPixelToShort_64x48_avx512);924
+ p.chroma[X265_CSP_I444].pu[LUMA_64x64].p2s[NONALIGNED] = PFX(filterPixelToShort_64x64_avx512);925
+926
+ p.pu[LUMA_64x16].convert_p2s[ALIGNED] = PFX(filterPixelToShort_aligned_64x16_avx512);927
+ p.pu[LUMA_64x32].convert_p2s[ALIGNED] = PFX(filterPixelToShort_aligned_64x32_avx512);928
+ p.pu[LUMA_64x48].convert_p2s[ALIGNED] = PFX(filterPixelToShort_aligned_64x48_avx512);929
+ p.pu[LUMA_64x64].convert_p2s[ALIGNED] = PFX(filterPixelToShort_aligned_64x64_avx512);930
+ p.pu[LUMA_32x8].convert_p2s[ALIGNED] = PFX(filterPixelToShort_aligned_32x8_avx512);931
+ p.pu[LUMA_32x16].convert_p2s[ALIGNED] = PFX(filterPixelToShort_aligned_32x16_avx512);932
+ p.pu[LUMA_32x24].convert_p2s[ALIGNED] = PFX(filterPixelToShort_aligned_32x24_avx512);933
+ p.pu[LUMA_32x32].convert_p2s[ALIGNED] = PFX(filterPixelToShort_aligned_32x32_avx512);934
+ p.pu[LUMA_32x64].convert_p2s[ALIGNED] = PFX(filterPixelToShort_aligned_32x64_avx512);935
+ p.pu[LUMA_48x64].convert_p2s[ALIGNED] = PFX(filterPixelToShort_aligned_48x64_avx512);936
+937
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x8].p2s[ALIGNED] = PFX(filterPixelToShort_aligned_32x8_avx512);938
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x16].p2s[ALIGNED] = PFX(filterPixelToShort_aligned_32x16_avx512);939
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x24].p2s[ALIGNED] = PFX(filterPixelToShort_aligned_32x24_avx512);940
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x32].p2s[ALIGNED] = PFX(filterPixelToShort_aligned_32x32_avx512);941
+942
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].p2s[ALIGNED] = PFX(filterPixelToShort_aligned_32x16_avx512);943
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x32].p2s[ALIGNED] = PFX(filterPixelToShort_aligned_32x32_avx512);944
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x48].p2s[ALIGNED] = PFX(filterPixelToShort_aligned_32x48_avx512);945
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x64].p2s[ALIGNED] = PFX(filterPixelToShort_aligned_32x64_avx512);946
+947
+ p.chroma[X265_CSP_I444].pu[LUMA_32x8].p2s[ALIGNED] = PFX(filterPixelToShort_aligned_32x8_avx512);948
+ p.chroma[X265_CSP_I444].pu[LUMA_32x16].p2s[ALIGNED] = PFX(filterPixelToShort_aligned_32x16_avx512);949
+ p.chroma[X265_CSP_I444].pu[LUMA_32x24].p2s[ALIGNED] = PFX(filterPixelToShort_aligned_32x24_avx512);950
+ p.chroma[X265_CSP_I444].pu[LUMA_32x32].p2s[ALIGNED] = PFX(filterPixelToShort_aligned_32x32_avx512);951
+ p.chroma[X265_CSP_I444].pu[LUMA_32x64].p2s[ALIGNED] = PFX(filterPixelToShort_aligned_32x64_avx512);952
+ p.chroma[X265_CSP_I444].pu[LUMA_64x16].p2s[ALIGNED] = PFX(filterPixelToShort_aligned_64x16_avx512);953
+ p.chroma[X265_CSP_I444].pu[LUMA_64x32].p2s[ALIGNED] = PFX(filterPixelToShort_aligned_64x32_avx512);954
+ p.chroma[X265_CSP_I444].pu[LUMA_64x48].p2s[ALIGNED] = PFX(filterPixelToShort_aligned_64x48_avx512);955
+ p.chroma[X265_CSP_I444].pu[LUMA_64x64].p2s[ALIGNED] = PFX(filterPixelToShort_aligned_64x64_avx512);956
+ p.cu[BLOCK_32x32].ssd_s[NONALIGNED] = PFX(pixel_ssd_s_32_avx512);957
+ p.cu[BLOCK_32x32].ssd_s[ALIGNED] = PFX(pixel_ssd_s_aligned_32_avx512);958
+ p.cu[BLOCK_16x16].ssd_s[NONALIGNED] = PFX(pixel_ssd_s_16_avx512);959
+ p.cu[BLOCK_16x16].ssd_s[ALIGNED] = PFX(pixel_ssd_s_aligned_16_avx512);960
+ p.pu[LUMA_16x32].sad = PFX(pixel_sad_16x32_avx512);961
+ p.pu[LUMA_16x64].sad = PFX(pixel_sad_16x64_avx512);962
+ p.pu[LUMA_32x8].sad = PFX(pixel_sad_32x8_avx512);963
+ p.pu[LUMA_32x16].sad = PFX(pixel_sad_32x16_avx512);964
+ p.pu[LUMA_32x24].sad = PFX(pixel_sad_32x24_avx512);965
+ p.pu[LUMA_32x32].sad = PFX(pixel_sad_32x32_avx512);966
+ p.pu[LUMA_32x64].sad = PFX(pixel_sad_32x64_avx512);967
+ p.pu[LUMA_48x64].sad = PFX(pixel_sad_48x64_avx512);968
+ p.pu[LUMA_64x16].sad = PFX(pixel_sad_64x16_avx512);969
+ p.pu[LUMA_64x32].sad = PFX(pixel_sad_64x32_avx512);970
+ p.pu[LUMA_64x48].sad = PFX(pixel_sad_64x48_avx512);971
+ p.pu[LUMA_64x64].sad = PFX(pixel_sad_64x64_avx512);972
+973
+ p.pu[LUMA_64x16].addAvg[NONALIGNED] = PFX(addAvg_64x16_avx512);974
+ p.pu[LUMA_64x32].addAvg[NONALIGNED] = PFX(addAvg_64x32_avx512);975
+ p.pu[LUMA_64x48].addAvg[NONALIGNED] = PFX(addAvg_64x48_avx512);976
+ p.pu[LUMA_64x64].addAvg[NONALIGNED] = PFX(addAvg_64x64_avx512);977
+ p.pu[LUMA_32x8].addAvg[NONALIGNED] = PFX(addAvg_32x8_avx512);978
+ p.pu[LUMA_32x16].addAvg[NONALIGNED] = PFX(addAvg_32x16_avx512);979
+ p.pu[LUMA_32x24].addAvg[NONALIGNED] = PFX(addAvg_32x24_avx512);980
+ p.pu[LUMA_32x32].addAvg[NONALIGNED] = PFX(addAvg_32x32_avx512);981
+ p.pu[LUMA_32x64].addAvg[NONALIGNED] = PFX(addAvg_32x64_avx512);982
+ p.pu[LUMA_16x4].addAvg[NONALIGNED] = PFX(addAvg_16x4_avx512);983
+ p.pu[LUMA_16x8].addAvg[NONALIGNED] = PFX(addAvg_16x8_avx512);984
+ p.pu[LUMA_16x12].addAvg[NONALIGNED] = PFX(addAvg_16x12_avx512);985
+ p.pu[LUMA_16x16].addAvg[NONALIGNED] = PFX(addAvg_16x16_avx512);986
+ p.pu[LUMA_16x32].addAvg[NONALIGNED] = PFX(addAvg_16x32_avx512);987
+ p.pu[LUMA_16x64].addAvg[NONALIGNED] = PFX(addAvg_16x64_avx512);988
+ p.pu[LUMA_48x64].addAvg[NONALIGNED] = PFX(addAvg_48x64_avx512);989
+990
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x8].addAvg[NONALIGNED] = PFX(addAvg_32x8_avx512);991
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x16].addAvg[NONALIGNED] = PFX(addAvg_32x16_avx512);992
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x24].addAvg[NONALIGNED] = PFX(addAvg_32x24_avx512);993
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x32].addAvg[NONALIGNED] = PFX(addAvg_32x32_avx512);994
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x4].addAvg[NONALIGNED] = PFX(addAvg_16x4_avx512);995
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x8].addAvg[NONALIGNED] = PFX(addAvg_16x8_avx512);996
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x12].addAvg[NONALIGNED] = PFX(addAvg_16x12_avx512);997
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x16].addAvg[NONALIGNED] = PFX(addAvg_16x16_avx512);998
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x32].addAvg[NONALIGNED] = PFX(addAvg_16x32_avx512);999
+1000
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].addAvg[NONALIGNED] = PFX(addAvg_32x16_avx512);1001
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x32].addAvg[NONALIGNED] = PFX(addAvg_32x32_avx512);1002
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x48].addAvg[NONALIGNED] = PFX(addAvg_32x48_avx512);1003
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x64].addAvg[NONALIGNED] = PFX(addAvg_32x64_avx512);1004
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x32].addAvg[NONALIGNED] = PFX(addAvg_16x32_avx512);1005
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x16].addAvg[NONALIGNED] = PFX(addAvg_16x16_avx512);1006
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x64].addAvg[NONALIGNED] = PFX(addAvg_16x64_avx512);1007
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x24].addAvg[NONALIGNED] = PFX(addAvg_16x24_avx512);1008
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x8].addAvg[NONALIGNED] = PFX(addAvg_16x8_avx512);1009
+ p.pu[LUMA_32x8].pixelavg_pp[NONALIGNED] = PFX(pixel_avg_32x8_avx512);1010
+ p.pu[LUMA_32x16].pixelavg_pp[NONALIGNED] = PFX(pixel_avg_32x16_avx512);1011
+ p.pu[LUMA_32x24].pixelavg_pp[NONALIGNED] = PFX(pixel_avg_32x24_avx512);1012
+ p.pu[LUMA_32x32].pixelavg_pp[NONALIGNED] = PFX(pixel_avg_32x32_avx512);1013
+ p.pu[LUMA_32x64].pixelavg_pp[NONALIGNED] = PFX(pixel_avg_32x64_avx512);1014
+ p.pu[LUMA_64x16].pixelavg_pp[NONALIGNED] = PFX(pixel_avg_64x16_avx512);1015
+ p.pu[LUMA_64x32].pixelavg_pp[NONALIGNED] = PFX(pixel_avg_64x32_avx512);1016
+ p.pu[LUMA_64x48].pixelavg_pp[NONALIGNED] = PFX(pixel_avg_64x48_avx512);1017
+ p.pu[LUMA_64x64].pixelavg_pp[NONALIGNED] = PFX(pixel_avg_64x64_avx512);1018
+ p.pu[LUMA_48x64].pixelavg_pp[NONALIGNED] = PFX(pixel_avg_48x64_avx512);1019
+1020
+ p.pu[LUMA_32x8].pixelavg_pp[ALIGNED] = PFX(pixel_avg_aligned_32x8_avx512);1021
+ p.pu[LUMA_32x16].pixelavg_pp[ALIGNED] = PFX(pixel_avg_aligned_32x16_avx512);1022
+ p.pu[LUMA_32x24].pixelavg_pp[ALIGNED] = PFX(pixel_avg_aligned_32x24_avx512);1023
+ p.pu[LUMA_32x32].pixelavg_pp[ALIGNED] = PFX(pixel_avg_aligned_32x32_avx512);1024
+ p.pu[LUMA_32x64].pixelavg_pp[ALIGNED] = PFX(pixel_avg_aligned_32x64_avx512);1025
+ p.pu[LUMA_48x64].pixelavg_pp[ALIGNED] = PFX(pixel_avg_aligned_48x64_avx512);1026
+ p.pu[LUMA_64x16].pixelavg_pp[ALIGNED] = PFX(pixel_avg_aligned_64x16_avx512);1027
+ p.pu[LUMA_64x32].pixelavg_pp[ALIGNED] = PFX(pixel_avg_aligned_64x32_avx512);1028
+ p.pu[LUMA_64x48].pixelavg_pp[ALIGNED] = PFX(pixel_avg_aligned_64x48_avx512);1029
+ p.pu[LUMA_64x64].pixelavg_pp[ALIGNED] = PFX(pixel_avg_aligned_64x64_avx512);1030
+ p.pu[LUMA_16x8].sad_x3 = PFX(pixel_sad_x3_16x8_avx512);1031
+ p.pu[LUMA_16x12].sad_x3 = PFX(pixel_sad_x3_16x12_avx512);1032
+ p.pu[LUMA_16x16].sad_x3 = PFX(pixel_sad_x3_16x16_avx512);1033
+ p.pu[LUMA_16x32].sad_x3 = PFX(pixel_sad_x3_16x32_avx512);1034
+ p.pu[LUMA_16x64].sad_x3 = PFX(pixel_sad_x3_16x64_avx512);1035
+ p.pu[LUMA_32x8].sad_x3 = PFX(pixel_sad_x3_32x8_avx512);1036
+ p.pu[LUMA_32x16].sad_x3 = PFX(pixel_sad_x3_32x16_avx512);1037
+ p.pu[LUMA_32x24].sad_x3 = PFX(pixel_sad_x3_32x24_avx512);1038
+ p.pu[LUMA_32x32].sad_x3 = PFX(pixel_sad_x3_32x32_avx512);1039
+ p.pu[LUMA_32x64].sad_x3 = PFX(pixel_sad_x3_32x64_avx512);1040
+ //p.pu[LUMA_48x64].sad_x3 = PFX(pixel_sad_x3_48x64_avx512);1041
+ p.pu[LUMA_64x16].sad_x3 = PFX(pixel_sad_x3_64x16_avx512);1042
+ p.pu[LUMA_64x32].sad_x3 = PFX(pixel_sad_x3_64x32_avx512);1043
+ p.pu[LUMA_64x48].sad_x3 = PFX(pixel_sad_x3_64x48_avx512);1044
+ p.pu[LUMA_64x64].sad_x3 = PFX(pixel_sad_x3_64x64_avx512);1045
+1046
+ p.pu[LUMA_16x8].sad_x4 = PFX(pixel_sad_x4_16x8_avx512);1047
+ p.pu[LUMA_16x12].sad_x4 = PFX(pixel_sad_x4_16x12_avx512);1048
+ p.pu[LUMA_16x16].sad_x4 = PFX(pixel_sad_x4_16x16_avx512);1049
+ p.pu[LUMA_16x32].sad_x4 = PFX(pixel_sad_x4_16x32_avx512);1050
+ p.pu[LUMA_16x64].sad_x4 = PFX(pixel_sad_x4_16x64_avx512);1051
+ p.pu[LUMA_32x8].sad_x4 = PFX(pixel_sad_x4_32x8_avx512);1052
+ p.pu[LUMA_32x16].sad_x4 = PFX(pixel_sad_x4_32x16_avx512);1053
+ p.pu[LUMA_32x24].sad_x4 = PFX(pixel_sad_x4_32x24_avx512);1054
+ p.pu[LUMA_32x32].sad_x4 = PFX(pixel_sad_x4_32x32_avx512);1055
+ p.pu[LUMA_32x64].sad_x4 = PFX(pixel_sad_x4_32x64_avx512);1056
+ //p.pu[LUMA_48x64].sad_x4 = PFX(pixel_sad_x4_48x64_avx512);1057
+ p.pu[LUMA_64x16].sad_x4 = PFX(pixel_sad_x4_64x16_avx512);1058
+ p.pu[LUMA_64x32].sad_x4 = PFX(pixel_sad_x4_64x32_avx512);1059
+ p.pu[LUMA_64x48].sad_x4 = PFX(pixel_sad_x4_64x48_avx512);1060
+ p.pu[LUMA_64x64].sad_x4 = PFX(pixel_sad_x4_64x64_avx512);1061
+ p.cu[BLOCK_16x16].cpy2Dto1D_shl = PFX(cpy2Dto1D_shl_16_avx512);1062
+ p.cu[BLOCK_32x32].cpy2Dto1D_shl = PFX(cpy2Dto1D_shl_32_avx512);1063
+ p.cu[BLOCK_32x32].cpy1Dto2D_shl[NONALIGNED] = PFX(cpy1Dto2D_shl_32_avx512);1064
+ p.cu[BLOCK_32x32].cpy1Dto2D_shl[ALIGNED] = PFX(cpy1Dto2D_shl_aligned_32_avx512);1065
+ p.cu[BLOCK_16x16].cpy1Dto2D_shr = PFX(cpy1Dto2D_shr_16_avx512);1066
+ p.cu[BLOCK_32x32].cpy1Dto2D_shr = PFX(cpy1Dto2D_shr_32_avx512);1067
+1068
+ p.cu[BLOCK_16x16].cpy2Dto1D_shr = PFX(cpy2Dto1D_shr_16_avx512);1069
+ p.cu[BLOCK_32x32].cpy2Dto1D_shr = PFX(cpy2Dto1D_shr_32_avx512);1070
+1071
+ p.weight_pp = PFX(weight_pp_avx512);1072
+ p.weight_sp = PFX(weight_sp_avx512);1073
+ p.dequant_normal = PFX(dequant_normal_avx512);1074
+ p.dequant_scaling = PFX(dequant_scaling_avx512);1075
+ p.cu[BLOCK_32x32].copy_cnt = PFX(copy_cnt_32_avx512);1076
+ p.cu[BLOCK_16x16].copy_cnt = PFX(copy_cnt_16_avx512);1077
+1078
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_8x4].filter_hpp = PFX(interp_4tap_horiz_pp_8x4_avx512);1079
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_8x8].filter_hpp = PFX(interp_4tap_horiz_pp_8x8_avx512);1080
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_8x16].filter_hpp = PFX(interp_4tap_horiz_pp_8x16_avx512);1081
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_8x32].filter_hpp = PFX(interp_4tap_horiz_pp_8x32_avx512);1082
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x4].filter_hpp = PFX(interp_4tap_horiz_pp_16x4_avx512);1083
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x8].filter_hpp = PFX(interp_4tap_horiz_pp_16x8_avx512);1084
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x12].filter_hpp = PFX(interp_4tap_horiz_pp_16x12_avx512);1085
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x16].filter_hpp = PFX(interp_4tap_horiz_pp_16x16_avx512);1086
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x32].filter_hpp = PFX(interp_4tap_horiz_pp_16x32_avx512);1087
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x8].filter_hpp = PFX(interp_4tap_horiz_pp_32x8_avx512);1088
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x16].filter_hpp = PFX(interp_4tap_horiz_pp_32x16_avx512);1089
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x24].filter_hpp = PFX(interp_4tap_horiz_pp_32x24_avx512);1090
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x32].filter_hpp = PFX(interp_4tap_horiz_pp_32x32_avx512);1091
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_24x32].filter_hpp = PFX(interp_4tap_horiz_pp_24x32_avx512);1092
+1093
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x4].filter_hpp = PFX(interp_4tap_horiz_pp_8x4_avx512);1094
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x8].filter_hpp = PFX(interp_4tap_horiz_pp_8x8_avx512);1095
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x12].filter_hpp = PFX(interp_4tap_horiz_pp_8x12_avx512);1096
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x16].filter_hpp = PFX(interp_4tap_horiz_pp_8x16_avx512);1097
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x32].filter_hpp = PFX(interp_4tap_horiz_pp_8x32_avx512);1098
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x64].filter_hpp = PFX(interp_4tap_horiz_pp_8x64_avx512);1099
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x8].filter_hpp = PFX(interp_4tap_horiz_pp_16x8_avx512);1100
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x16].filter_hpp = PFX(interp_4tap_horiz_pp_16x16_avx512);1101
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x24].filter_hpp = PFX(interp_4tap_horiz_pp_16x24_avx512);1102
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x32].filter_hpp = PFX(interp_4tap_horiz_pp_16x32_avx512);1103
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x64].filter_hpp = PFX(interp_4tap_horiz_pp_16x64_avx512);1104
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].filter_hpp = PFX(interp_4tap_horiz_pp_32x16_avx512);1105
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x32].filter_hpp = PFX(interp_4tap_horiz_pp_32x32_avx512);1106
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x48].filter_hpp = PFX(interp_4tap_horiz_pp_32x48_avx512);1107
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x64].filter_hpp = PFX(interp_4tap_horiz_pp_32x64_avx512);1108
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_24x64].filter_hpp = PFX(interp_4tap_horiz_pp_24x64_avx512);1109
+1110
+ p.chroma[X265_CSP_I444].pu[LUMA_8x4].filter_hpp = PFX(interp_4tap_horiz_pp_8x4_avx512);1111
+ p.chroma[X265_CSP_I444].pu[LUMA_8x8].filter_hpp = PFX(interp_4tap_horiz_pp_8x8_avx512);1112
+ p.chroma[X265_CSP_I444].pu[LUMA_8x16].filter_hpp = PFX(interp_4tap_horiz_pp_8x16_avx512);1113
+ p.chroma[X265_CSP_I444].pu[LUMA_8x32].filter_hpp = PFX(interp_4tap_horiz_pp_8x32_avx512);1114
+ p.chroma[X265_CSP_I444].pu[LUMA_16x4].filter_hpp = PFX(interp_4tap_horiz_pp_16x4_avx512);1115
+ p.chroma[X265_CSP_I444].pu[LUMA_16x8].filter_hpp = PFX(interp_4tap_horiz_pp_16x8_avx512);1116
+ p.chroma[X265_CSP_I444].pu[LUMA_16x12].filter_hpp = PFX(interp_4tap_horiz_pp_16x12_avx512);1117
+ p.chroma[X265_CSP_I444].pu[LUMA_16x16].filter_hpp = PFX(interp_4tap_horiz_pp_16x16_avx512);1118
+ p.chroma[X265_CSP_I444].pu[LUMA_16x32].filter_hpp = PFX(interp_4tap_horiz_pp_16x32_avx512);1119
+ p.chroma[X265_CSP_I444].pu[LUMA_16x64].filter_hpp = PFX(interp_4tap_horiz_pp_16x64_avx512);1120
+ p.chroma[X265_CSP_I444].pu[LUMA_32x8].filter_hpp = PFX(interp_4tap_horiz_pp_32x8_avx512);1121
+ p.chroma[X265_CSP_I444].pu[LUMA_32x16].filter_hpp = PFX(interp_4tap_horiz_pp_32x16_avx512);1122
+ p.chroma[X265_CSP_I444].pu[LUMA_32x24].filter_hpp = PFX(interp_4tap_horiz_pp_32x24_avx512);1123
+ p.chroma[X265_CSP_I444].pu[LUMA_32x32].filter_hpp = PFX(interp_4tap_horiz_pp_32x32_avx512);1124
+ p.chroma[X265_CSP_I444].pu[LUMA_32x64].filter_hpp = PFX(interp_4tap_horiz_pp_32x64_avx512);1125
+ p.chroma[X265_CSP_I444].pu[LUMA_64x16].filter_hpp = PFX(interp_4tap_horiz_pp_64x16_avx512);1126
+ p.chroma[X265_CSP_I444].pu[LUMA_64x32].filter_hpp = PFX(interp_4tap_horiz_pp_64x32_avx512);1127
+ p.chroma[X265_CSP_I444].pu[LUMA_64x48].filter_hpp = PFX(interp_4tap_horiz_pp_64x48_avx512);1128
+ p.chroma[X265_CSP_I444].pu[LUMA_64x64].filter_hpp = PFX(interp_4tap_horiz_pp_64x64_avx512);1129
+ p.chroma[X265_CSP_I444].pu[LUMA_48x64].filter_hpp = PFX(interp_4tap_horiz_pp_48x64_avx512);1130
+ p.chroma[X265_CSP_I444].pu[LUMA_24x32].filter_hpp = PFX(interp_4tap_horiz_pp_24x32_avx512);1131
+1132
+ p.pu[LUMA_16x4].addAvg[ALIGNED] = PFX(addAvg_aligned_16x4_avx512);1133
+ p.pu[LUMA_16x8].addAvg[ALIGNED] = PFX(addAvg_aligned_16x8_avx512);1134
+ p.pu[LUMA_16x12].addAvg[ALIGNED] = PFX(addAvg_aligned_16x12_avx512);1135
+ p.pu[LUMA_16x16].addAvg[ALIGNED] = PFX(addAvg_aligned_16x16_avx512);1136
+ p.pu[LUMA_16x32].addAvg[ALIGNED] = PFX(addAvg_aligned_16x32_avx512);1137
+ p.pu[LUMA_16x64].addAvg[ALIGNED] = PFX(addAvg_aligned_16x64_avx512);1138
+ p.pu[LUMA_48x64].addAvg[ALIGNED] = PFX(addAvg_aligned_48x64_avx512);1139
+ p.pu[LUMA_32x8].addAvg[ALIGNED] = PFX(addAvg_aligned_32x8_avx512);1140
+ p.pu[LUMA_32x16].addAvg[ALIGNED] = PFX(addAvg_aligned_32x16_avx512);1141
+ p.pu[LUMA_32x24].addAvg[ALIGNED] = PFX(addAvg_aligned_32x24_avx512);1142
+ p.pu[LUMA_32x32].addAvg[ALIGNED] = PFX(addAvg_aligned_32x32_avx512);1143
+ p.pu[LUMA_32x64].addAvg[ALIGNED] = PFX(addAvg_aligned_32x64_avx512);1144
+ p.pu[LUMA_64x16].addAvg[ALIGNED] = PFX(addAvg_aligned_64x16_avx512);1145
+ p.pu[LUMA_64x32].addAvg[ALIGNED] = PFX(addAvg_aligned_64x32_avx512);1146
+ p.pu[LUMA_64x48].addAvg[ALIGNED] = PFX(addAvg_aligned_64x48_avx512);1147
+ p.pu[LUMA_64x64].addAvg[ALIGNED] = PFX(addAvg_aligned_64x64_avx512);1148
+1149
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x4].addAvg[ALIGNED] = PFX(addAvg_aligned_16x4_avx512);1150
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x8].addAvg[ALIGNED] = PFX(addAvg_aligned_16x8_avx512);1151
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x12].addAvg[ALIGNED] = PFX(addAvg_aligned_16x12_avx512);1152
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x16].addAvg[ALIGNED] = PFX(addAvg_aligned_16x16_avx512);1153
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x32].addAvg[ALIGNED] = PFX(addAvg_aligned_16x32_avx512);1154
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x8].addAvg[ALIGNED] = PFX(addAvg_aligned_32x8_avx512);1155
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x16].addAvg[ALIGNED] = PFX(addAvg_aligned_32x16_avx512);1156
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x24].addAvg[ALIGNED] = PFX(addAvg_aligned_32x24_avx512);1157
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x32].addAvg[ALIGNED] = PFX(addAvg_aligned_32x32_avx512);1158
+1159
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x32].addAvg[ALIGNED] = PFX(addAvg_aligned_16x32_avx512);1160
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x16].addAvg[ALIGNED] = PFX(addAvg_aligned_16x16_avx512);1161
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x64].addAvg[ALIGNED] = PFX(addAvg_aligned_16x64_avx512);1162
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x24].addAvg[ALIGNED] = PFX(addAvg_aligned_16x24_avx512);1163
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x8].addAvg[ALIGNED] = PFX(addAvg_aligned_16x8_avx512);1164
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].addAvg[ALIGNED] = PFX(addAvg_aligned_32x16_avx512);1165
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x32].addAvg[ALIGNED] = PFX(addAvg_aligned_32x32_avx512);1166
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x48].addAvg[ALIGNED] = PFX(addAvg_aligned_32x48_avx512);1167
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x64].addAvg[ALIGNED] = PFX(addAvg_aligned_32x64_avx512);1168
+ p.cu[BLOCK_32x32].blockfill_s[NONALIGNED] = PFX(blockfill_s_32x32_avx512);1169
+ p.cu[BLOCK_32x32].blockfill_s[ALIGNED] = PFX(blockfill_s_aligned_32x32_avx512);1170
+ p.pu[LUMA_8x4].luma_hpp = PFX(interp_8tap_horiz_pp_8x4_avx512);1171
+ p.pu[LUMA_8x8].luma_hpp = PFX(interp_8tap_horiz_pp_8x8_avx512);1172
+ p.pu[LUMA_8x16].luma_hpp = PFX(interp_8tap_horiz_pp_8x16_avx512);1173
+ p.pu[LUMA_8x32].luma_hpp = PFX(interp_8tap_horiz_pp_8x32_avx512);1174
+ p.pu[LUMA_16x4].luma_hpp = PFX(interp_8tap_horiz_pp_16x4_avx512);1175
+ p.pu[LUMA_16x8].luma_hpp = PFX(interp_8tap_horiz_pp_16x8_avx512);1176
+ p.pu[LUMA_16x12].luma_hpp = PFX(interp_8tap_horiz_pp_16x12_avx512);1177
+ p.pu[LUMA_16x16].luma_hpp = PFX(interp_8tap_horiz_pp_16x16_avx512);1178
+ p.pu[LUMA_16x32].luma_hpp = PFX(interp_8tap_horiz_pp_16x32_avx512);1179
+ p.pu[LUMA_16x64].luma_hpp = PFX(interp_8tap_horiz_pp_16x64_avx512);1180
+ p.pu[LUMA_24x32].luma_hpp = PFX(interp_8tap_horiz_pp_24x32_avx512);1181
+ p.pu[LUMA_32x8].luma_hpp = PFX(interp_8tap_horiz_pp_32x8_avx512);1182
+ p.pu[LUMA_32x16].luma_hpp = PFX(interp_8tap_horiz_pp_32x16_avx512);1183
+ p.pu[LUMA_32x24].luma_hpp = PFX(interp_8tap_horiz_pp_32x24_avx512);1184
+ p.pu[LUMA_32x32].luma_hpp = PFX(interp_8tap_horiz_pp_32x32_avx512);1185
+ p.pu[LUMA_32x64].luma_hpp = PFX(interp_8tap_horiz_pp_32x64_avx512);1186
+ p.pu[LUMA_64x16].luma_hpp = PFX(interp_8tap_horiz_pp_64x16_avx512);1187
+ p.pu[LUMA_64x32].luma_hpp = PFX(interp_8tap_horiz_pp_64x32_avx512);1188
+ p.pu[LUMA_64x48].luma_hpp = PFX(interp_8tap_horiz_pp_64x48_avx512);1189
+ p.pu[LUMA_64x64].luma_hpp = PFX(interp_8tap_horiz_pp_64x64_avx512);1190
+ p.pu[LUMA_48x64].luma_hpp = PFX(interp_8tap_horiz_pp_48x64_avx512);1191
+1192
+ p.chroma[X265_CSP_I444].pu[LUMA_64x16].filter_vpp = PFX(interp_4tap_vert_pp_64x16_avx512);1193
+ p.chroma[X265_CSP_I444].pu[LUMA_64x32].filter_vpp = PFX(interp_4tap_vert_pp_64x32_avx512);1194
+ p.chroma[X265_CSP_I444].pu[LUMA_64x48].filter_vpp = PFX(interp_4tap_vert_pp_64x48_avx512);1195
+ p.chroma[X265_CSP_I444].pu[LUMA_64x64].filter_vpp = PFX(interp_4tap_vert_pp_64x64_avx512);1196
+ p.chroma[X265_CSP_I444].pu[LUMA_64x16].filter_vps = PFX(interp_4tap_vert_ps_64x16_avx512);1197
+ p.chroma[X265_CSP_I444].pu[LUMA_64x32].filter_vps = PFX(interp_4tap_vert_ps_64x32_avx512);1198
+ p.chroma[X265_CSP_I444].pu[LUMA_64x48].filter_vps = PFX(interp_4tap_vert_ps_64x48_avx512);1199
+ p.chroma[X265_CSP_I444].pu[LUMA_64x64].filter_vps = PFX(interp_4tap_vert_ps_64x64_avx512);1200
+ p.chroma[X265_CSP_I444].pu[LUMA_64x16].filter_vsp = PFX(interp_4tap_vert_sp_64x16_avx512);1201
+ p.chroma[X265_CSP_I444].pu[LUMA_64x32].filter_vsp = PFX(interp_4tap_vert_sp_64x32_avx512);1202
+ p.chroma[X265_CSP_I444].pu[LUMA_64x48].filter_vsp = PFX(interp_4tap_vert_sp_64x48_avx512);1203
+ p.chroma[X265_CSP_I444].pu[LUMA_64x64].filter_vsp = PFX(interp_4tap_vert_sp_64x64_avx512);1204
+ p.chroma[X265_CSP_I444].pu[LUMA_64x16].filter_vss = PFX(interp_4tap_vert_ss_64x16_avx512);1205
+ p.chroma[X265_CSP_I444].pu[LUMA_64x32].filter_vss = PFX(interp_4tap_vert_ss_64x32_avx512);1206
+ p.chroma[X265_CSP_I444].pu[LUMA_64x48].filter_vss = PFX(interp_4tap_vert_ss_64x48_avx512);1207
+ p.chroma[X265_CSP_I444].pu[LUMA_64x64].filter_vss = PFX(interp_4tap_vert_ss_64x64_avx512);1208
+1209
+ p.chroma[X265_CSP_I444].pu[LUMA_48x64].filter_vpp = PFX(interp_4tap_vert_pp_48x64_avx512);1210
+ p.chroma[X265_CSP_I444].pu[LUMA_48x64].filter_vps = PFX(interp_4tap_vert_ps_48x64_avx512);1211
+ p.chroma[X265_CSP_I444].pu[LUMA_48x64].filter_vsp = PFX(interp_4tap_vert_sp_48x64_avx512);1212
+ p.chroma[X265_CSP_I444].pu[LUMA_48x64].filter_vss = PFX(interp_4tap_vert_ss_48x64_avx512);1213
+1214
+ p.chroma[X265_CSP_I444].pu[LUMA_32x8].filter_vpp = PFX(interp_4tap_vert_pp_32x8_avx512);1215
+ p.chroma[X265_CSP_I444].pu[LUMA_32x16].filter_vpp = PFX(interp_4tap_vert_pp_32x16_avx512);1216
+ p.chroma[X265_CSP_I444].pu[LUMA_32x24].filter_vpp = PFX(interp_4tap_vert_pp_32x24_avx512);1217
+ p.chroma[X265_CSP_I444].pu[LUMA_32x32].filter_vpp = PFX(interp_4tap_vert_pp_32x32_avx512);1218
+ p.chroma[X265_CSP_I444].pu[LUMA_32x64].filter_vpp = PFX(interp_4tap_vert_pp_32x64_avx512);1219
+ p.chroma[X265_CSP_I444].pu[LUMA_32x8].filter_vps = PFX(interp_4tap_vert_ps_32x8_avx512);1220
+ p.chroma[X265_CSP_I444].pu[LUMA_32x16].filter_vps = PFX(interp_4tap_vert_ps_32x16_avx512);1221
+ p.chroma[X265_CSP_I444].pu[LUMA_32x24].filter_vps = PFX(interp_4tap_vert_ps_32x24_avx512);1222
+ p.chroma[X265_CSP_I444].pu[LUMA_32x32].filter_vps = PFX(interp_4tap_vert_ps_32x32_avx512);1223
+ p.chroma[X265_CSP_I444].pu[LUMA_32x64].filter_vps = PFX(interp_4tap_vert_ps_32x64_avx512);1224
+ p.chroma[X265_CSP_I444].pu[LUMA_32x8].filter_vss = PFX(interp_4tap_vert_ss_32x8_avx512);1225
+ p.chroma[X265_CSP_I444].pu[LUMA_32x16].filter_vss = PFX(interp_4tap_vert_ss_32x16_avx512);1226
+ p.chroma[X265_CSP_I444].pu[LUMA_32x24].filter_vss = PFX(interp_4tap_vert_ss_32x24_avx512);1227
+ p.chroma[X265_CSP_I444].pu[LUMA_32x32].filter_vss = PFX(interp_4tap_vert_ss_32x32_avx512);1228
+ p.chroma[X265_CSP_I444].pu[LUMA_32x64].filter_vss = PFX(interp_4tap_vert_ss_32x64_avx512);1229
+ p.chroma[X265_CSP_I444].pu[LUMA_32x8].filter_vsp = PFX(interp_4tap_vert_sp_32x8_avx512);1230
+ p.chroma[X265_CSP_I444].pu[LUMA_32x16].filter_vsp = PFX(interp_4tap_vert_sp_32x16_avx512);1231
+ p.chroma[X265_CSP_I444].pu[LUMA_32x24].filter_vsp = PFX(interp_4tap_vert_sp_32x24_avx512);1232
+ p.chroma[X265_CSP_I444].pu[LUMA_32x32].filter_vsp = PFX(interp_4tap_vert_sp_32x32_avx512);1233
+ p.chroma[X265_CSP_I444].pu[LUMA_32x64].filter_vsp = PFX(interp_4tap_vert_sp_32x64_avx512);1234
+1235
+ p.chroma[X265_CSP_I444].pu[LUMA_16x4].filter_vpp = PFX(interp_4tap_vert_pp_16x4_avx512);1236
+ p.chroma[X265_CSP_I444].pu[LUMA_16x8].filter_vpp = PFX(interp_4tap_vert_pp_16x8_avx512);1237
+ p.chroma[X265_CSP_I444].pu[LUMA_16x12].filter_vpp = PFX(interp_4tap_vert_pp_16x12_avx512);1238
+ p.chroma[X265_CSP_I444].pu[LUMA_16x16].filter_vpp = PFX(interp_4tap_vert_pp_16x16_avx512);1239
+ p.chroma[X265_CSP_I444].pu[LUMA_16x32].filter_vpp = PFX(interp_4tap_vert_pp_16x32_avx512);1240
+ p.chroma[X265_CSP_I444].pu[LUMA_16x64].filter_vpp = PFX(interp_4tap_vert_pp_16x64_avx512);1241
+ p.chroma[X265_CSP_I444].pu[LUMA_16x4].filter_vps = PFX(interp_4tap_vert_ps_16x4_avx512);1242
+ p.chroma[X265_CSP_I444].pu[LUMA_16x8].filter_vps = PFX(interp_4tap_vert_ps_16x8_avx512);1243
+ p.chroma[X265_CSP_I444].pu[LUMA_16x12].filter_vps = PFX(interp_4tap_vert_ps_16x12_avx512);1244
+ p.chroma[X265_CSP_I444].pu[LUMA_16x16].filter_vps = PFX(interp_4tap_vert_ps_16x16_avx512);1245
+ p.chroma[X265_CSP_I444].pu[LUMA_16x32].filter_vps = PFX(interp_4tap_vert_ps_16x32_avx512);1246
+ p.chroma[X265_CSP_I444].pu[LUMA_16x64].filter_vps = PFX(interp_4tap_vert_ps_16x64_avx512);1247
+ p.chroma[X265_CSP_I444].pu[LUMA_16x4].filter_vss = PFX(interp_4tap_vert_ss_16x4_avx512);1248
+ p.chroma[X265_CSP_I444].pu[LUMA_16x8].filter_vss = PFX(interp_4tap_vert_ss_16x8_avx512);1249
+ p.chroma[X265_CSP_I444].pu[LUMA_16x12].filter_vss = PFX(interp_4tap_vert_ss_16x12_avx512);1250
+ p.chroma[X265_CSP_I444].pu[LUMA_16x16].filter_vss = PFX(interp_4tap_vert_ss_16x16_avx512);1251
+ p.chroma[X265_CSP_I444].pu[LUMA_16x32].filter_vss = PFX(interp_4tap_vert_ss_16x32_avx512);1252
+ p.chroma[X265_CSP_I444].pu[LUMA_16x64].filter_vss = PFX(interp_4tap_vert_ss_16x64_avx512);1253
+ p.chroma[X265_CSP_I444].pu[LUMA_16x4].filter_vsp = PFX(interp_4tap_vert_sp_16x4_avx512);1254
+ p.chroma[X265_CSP_I444].pu[LUMA_16x8].filter_vsp = PFX(interp_4tap_vert_sp_16x8_avx512);1255
+ p.chroma[X265_CSP_I444].pu[LUMA_16x12].filter_vsp = PFX(interp_4tap_vert_sp_16x12_avx512);1256
+ p.chroma[X265_CSP_I444].pu[LUMA_16x16].filter_vsp = PFX(interp_4tap_vert_sp_16x16_avx512);1257
+ p.chroma[X265_CSP_I444].pu[LUMA_16x32].filter_vsp = PFX(interp_4tap_vert_sp_16x32_avx512);1258
+ p.chroma[X265_CSP_I444].pu[LUMA_16x64].filter_vsp = PFX(interp_4tap_vert_sp_16x64_avx512);1259
+1260
+ p.chroma[X265_CSP_I444].pu[LUMA_8x8].filter_vpp = PFX(interp_4tap_vert_pp_8x8_avx512);1261
+ p.chroma[X265_CSP_I444].pu[LUMA_8x16].filter_vpp = PFX(interp_4tap_vert_pp_8x16_avx512);1262
+ p.chroma[X265_CSP_I444].pu[LUMA_8x32].filter_vpp = PFX(interp_4tap_vert_pp_8x32_avx512);1263
+ p.chroma[X265_CSP_I444].pu[LUMA_8x8].filter_vps = PFX(interp_4tap_vert_ps_8x8_avx512);1264
+ p.chroma[X265_CSP_I444].pu[LUMA_8x16].filter_vps = PFX(interp_4tap_vert_ps_8x16_avx512);1265
+ p.chroma[X265_CSP_I444].pu[LUMA_8x32].filter_vps = PFX(interp_4tap_vert_ps_8x32_avx512);1266
+ p.chroma[X265_CSP_I444].pu[LUMA_8x8].filter_vss = PFX(interp_4tap_vert_ss_8x8_avx512);1267
+ p.chroma[X265_CSP_I444].pu[LUMA_8x16].filter_vss = PFX(interp_4tap_vert_ss_8x16_avx512);1268
+ p.chroma[X265_CSP_I444].pu[LUMA_8x32].filter_vss = PFX(interp_4tap_vert_ss_8x32_avx512);1269
+ p.chroma[X265_CSP_I444].pu[LUMA_8x8].filter_vsp = PFX(interp_4tap_vert_sp_8x8_avx512);1270
+ p.chroma[X265_CSP_I444].pu[LUMA_8x16].filter_vsp = PFX(interp_4tap_vert_sp_8x16_avx512);1271
+ p.chroma[X265_CSP_I444].pu[LUMA_8x32].filter_vsp = PFX(interp_4tap_vert_sp_8x32_avx512);1272
+1273
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].filter_vpp = PFX(interp_4tap_vert_pp_32x16_avx512);1274
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x32].filter_vpp = PFX(interp_4tap_vert_pp_32x32_avx512);1275
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x48].filter_vpp = PFX(interp_4tap_vert_pp_32x48_avx512);1276
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x64].filter_vpp = PFX(interp_4tap_vert_pp_32x64_avx512);1277
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].filter_vps = PFX(interp_4tap_vert_ps_32x16_avx512);1278
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x32].filter_vps = PFX(interp_4tap_vert_ps_32x32_avx512);1279
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x48].filter_vps = PFX(interp_4tap_vert_ps_32x48_avx512);1280
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x64].filter_vps = PFX(interp_4tap_vert_ps_32x64_avx512);1281
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].filter_vss = PFX(interp_4tap_vert_ss_32x16_avx512);1282
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x32].filter_vss = PFX(interp_4tap_vert_ss_32x32_avx512);1283
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x48].filter_vss = PFX(interp_4tap_vert_ss_32x48_avx512);1284
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x64].filter_vss = PFX(interp_4tap_vert_ss_32x64_avx512);1285
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].filter_vsp = PFX(interp_4tap_vert_sp_32x16_avx512);1286
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x32].filter_vsp = PFX(interp_4tap_vert_sp_32x32_avx512);1287
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x48].filter_vsp = PFX(interp_4tap_vert_sp_32x48_avx512);1288
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x64].filter_vsp = PFX(interp_4tap_vert_sp_32x64_avx512);1289
+1290
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x8].filter_vpp = PFX(interp_4tap_vert_pp_16x8_avx512);1291
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x16].filter_vpp = PFX(interp_4tap_vert_pp_16x16_avx512);1292
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x24].filter_vpp = PFX(interp_4tap_vert_pp_16x24_avx512);1293
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x32].filter_vpp = PFX(interp_4tap_vert_pp_16x32_avx512);1294
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x64].filter_vpp = PFX(interp_4tap_vert_pp_16x64_avx512);1295
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x8].filter_vps = PFX(interp_4tap_vert_ps_16x8_avx512);1296
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x16].filter_vps = PFX(interp_4tap_vert_ps_16x16_avx512);1297
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x24].filter_vps = PFX(interp_4tap_vert_ps_16x24_avx512);1298
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x32].filter_vps = PFX(interp_4tap_vert_ps_16x32_avx512);1299
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x64].filter_vps = PFX(interp_4tap_vert_ps_16x64_avx512);1300
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x8].filter_vss = PFX(interp_4tap_vert_ss_16x8_avx512);1301
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x16].filter_vss = PFX(interp_4tap_vert_ss_16x16_avx512);1302
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x24].filter_vss = PFX(interp_4tap_vert_ss_16x24_avx512);1303
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x32].filter_vss = PFX(interp_4tap_vert_ss_16x32_avx512);1304
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x64].filter_vss = PFX(interp_4tap_vert_ss_16x64_avx512);1305
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x8].filter_vsp = PFX(interp_4tap_vert_sp_16x8_avx512);1306
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x16].filter_vsp = PFX(interp_4tap_vert_sp_16x16_avx512);1307
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x24].filter_vsp = PFX(interp_4tap_vert_sp_16x24_avx512);1308
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x32].filter_vsp = PFX(interp_4tap_vert_sp_16x32_avx512);1309
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x64].filter_vsp = PFX(interp_4tap_vert_sp_16x64_avx512);1310
+1311
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x8].filter_vpp = PFX(interp_4tap_vert_pp_8x8_avx512);1312
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x16].filter_vpp = PFX(interp_4tap_vert_pp_8x16_avx512);1313
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x32].filter_vpp = PFX(interp_4tap_vert_pp_8x32_avx512);1314
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x64].filter_vpp = PFX(interp_4tap_vert_pp_8x64_avx512);1315
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x8].filter_vps = PFX(interp_4tap_vert_ps_8x8_avx512);1316
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x16].filter_vps = PFX(interp_4tap_vert_ps_8x16_avx512);1317
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x32].filter_vps = PFX(interp_4tap_vert_ps_8x32_avx512);1318
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x64].filter_vps = PFX(interp_4tap_vert_ps_8x64_avx512);1319
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x8].filter_vss = PFX(interp_4tap_vert_ss_8x8_avx512);1320
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x16].filter_vss = PFX(interp_4tap_vert_ss_8x16_avx512);1321
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x32].filter_vss = PFX(interp_4tap_vert_ss_8x32_avx512);1322
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x64].filter_vss = PFX(interp_4tap_vert_ss_8x64_avx512);1323
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x8].filter_vsp = PFX(interp_4tap_vert_sp_8x8_avx512);1324
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x16].filter_vsp = PFX(interp_4tap_vert_sp_8x16_avx512);1325
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x32].filter_vsp = PFX(interp_4tap_vert_sp_8x32_avx512);1326
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x64].filter_vsp = PFX(interp_4tap_vert_sp_8x64_avx512);1327
+1328
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x8].filter_vpp = PFX(interp_4tap_vert_pp_32x8_avx512);1329
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x16].filter_vpp = PFX(interp_4tap_vert_pp_32x16_avx512);1330
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x24].filter_vpp = PFX(interp_4tap_vert_pp_32x24_avx512);1331
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x32].filter_vpp = PFX(interp_4tap_vert_pp_32x32_avx512);1332
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x8].filter_vps = PFX(interp_4tap_vert_ps_32x8_avx512);1333
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x16].filter_vps = PFX(interp_4tap_vert_ps_32x16_avx512);1334
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x24].filter_vps = PFX(interp_4tap_vert_ps_32x24_avx512);1335
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x32].filter_vps = PFX(interp_4tap_vert_ps_32x32_avx512);1336
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x8].filter_vss = PFX(interp_4tap_vert_ss_32x8_avx512);1337
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x16].filter_vss = PFX(interp_4tap_vert_ss_32x16_avx512);1338
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x24].filter_vss = PFX(interp_4tap_vert_ss_32x24_avx512);1339
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x32].filter_vss = PFX(interp_4tap_vert_ss_32x32_avx512);1340
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x8].filter_vsp = PFX(interp_4tap_vert_sp_32x8_avx512);1341
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x16].filter_vsp = PFX(interp_4tap_vert_sp_32x16_avx512);1342
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x24].filter_vsp = PFX(interp_4tap_vert_sp_32x24_avx512);1343
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x32].filter_vsp = PFX(interp_4tap_vert_sp_32x32_avx512);1344
+1345
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x4].filter_vpp = PFX(interp_4tap_vert_pp_16x4_avx512);1346
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x8].filter_vpp = PFX(interp_4tap_vert_pp_16x8_avx512);1347
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x12].filter_vpp = PFX(interp_4tap_vert_pp_16x12_avx512);1348
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x16].filter_vpp = PFX(interp_4tap_vert_pp_16x16_avx512);1349
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x32].filter_vpp = PFX(interp_4tap_vert_pp_16x32_avx512);1350
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x4].filter_vps = PFX(interp_4tap_vert_ps_16x4_avx512);1351
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x8].filter_vps = PFX(interp_4tap_vert_ps_16x8_avx512);1352
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x12].filter_vps = PFX(interp_4tap_vert_ps_16x12_avx512);1353
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x16].filter_vps = PFX(interp_4tap_vert_ps_16x16_avx512);1354
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x32].filter_vps = PFX(interp_4tap_vert_ps_16x32_avx512);1355
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x4].filter_vss = PFX(interp_4tap_vert_ss_16x4_avx512);1356
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x8].filter_vss = PFX(interp_4tap_vert_ss_16x8_avx512);1357
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x12].filter_vss = PFX(interp_4tap_vert_ss_16x12_avx512);1358
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x16].filter_vss = PFX(interp_4tap_vert_ss_16x16_avx512);1359
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x32].filter_vss = PFX(interp_4tap_vert_ss_16x32_avx512);1360
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x4].filter_vsp = PFX(interp_4tap_vert_sp_16x4_avx512);1361
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x8].filter_vsp = PFX(interp_4tap_vert_sp_16x8_avx512);1362
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x12].filter_vsp = PFX(interp_4tap_vert_sp_16x12_avx512);1363
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x16].filter_vsp = PFX(interp_4tap_vert_sp_16x16_avx512);1364
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x32].filter_vsp = PFX(interp_4tap_vert_sp_16x32_avx512);1365
+1366
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_8x8].filter_vpp = PFX(interp_4tap_vert_pp_8x8_avx512);1367
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_8x16].filter_vpp = PFX(interp_4tap_vert_pp_8x16_avx512);1368
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_8x32].filter_vpp = PFX(interp_4tap_vert_pp_8x32_avx512);1369
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_8x8].filter_vps = PFX(interp_4tap_vert_ps_8x8_avx512);1370
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_8x16].filter_vps = PFX(interp_4tap_vert_ps_8x16_avx512);1371
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_8x32].filter_vps = PFX(interp_4tap_vert_ps_8x32_avx512);1372
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_8x8].filter_vss = PFX(interp_4tap_vert_ss_8x8_avx512);1373
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_8x16].filter_vss = PFX(interp_4tap_vert_ss_8x16_avx512);1374
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_8x32].filter_vss = PFX(interp_4tap_vert_ss_8x32_avx512);1375
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_8x8].filter_vsp = PFX(interp_4tap_vert_sp_8x8_avx512);1376
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_8x16].filter_vsp = PFX(interp_4tap_vert_sp_8x16_avx512);1377
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_8x32].filter_vsp = PFX(interp_4tap_vert_sp_8x32_avx512);1378
+1379
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_24x32].filter_vpp = PFX(interp_4tap_vert_pp_24x32_avx512);1380
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_24x64].filter_vpp = PFX(interp_4tap_vert_pp_24x64_avx512);1381
+ p.chroma[X265_CSP_I444].pu[LUMA_24x32].filter_vpp = PFX(interp_4tap_vert_pp_24x32_avx512);1382
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_24x32].filter_vps = PFX(interp_4tap_vert_ps_24x32_avx512);1383
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_24x64].filter_vps = PFX(interp_4tap_vert_ps_24x64_avx512);1384
+ p.chroma[X265_CSP_I444].pu[LUMA_24x32].filter_vps = PFX(interp_4tap_vert_ps_24x32_avx512);1385
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_24x32].filter_vss = PFX(interp_4tap_vert_ss_24x32_avx512);1386
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_24x64].filter_vss = PFX(interp_4tap_vert_ss_24x64_avx512);1387
+ p.chroma[X265_CSP_I444].pu[LUMA_24x32].filter_vss = PFX(interp_4tap_vert_ss_24x32_avx512);1388
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_24x32].filter_vsp = PFX(interp_4tap_vert_sp_24x32_avx512);1389
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_24x64].filter_vsp = PFX(interp_4tap_vert_sp_24x64_avx512);1390
+ p.chroma[X265_CSP_I444].pu[LUMA_24x32].filter_vsp = PFX(interp_4tap_vert_sp_24x32_avx512);1391
+1392
+ p.pu[LUMA_8x8].luma_vss = PFX(interp_8tap_vert_ss_8x8_avx512);1393
+ p.pu[LUMA_8x16].luma_vss = PFX(interp_8tap_vert_ss_8x16_avx512);1394
+ p.pu[LUMA_8x32].luma_vss = PFX(interp_8tap_vert_ss_8x32_avx512);1395
+ p.pu[LUMA_16x4].luma_vss = PFX(interp_8tap_vert_ss_16x4_avx512);1396
+ p.pu[LUMA_16x8].luma_vss = PFX(interp_8tap_vert_ss_16x8_avx512);1397
+ p.pu[LUMA_16x12].luma_vss = PFX(interp_8tap_vert_ss_16x12_avx512);1398
+ p.pu[LUMA_16x16].luma_vss = PFX(interp_8tap_vert_ss_16x16_avx512);1399
+ p.pu[LUMA_16x32].luma_vss = PFX(interp_8tap_vert_ss_16x32_avx512);1400
+ p.pu[LUMA_16x64].luma_vss = PFX(interp_8tap_vert_ss_16x64_avx512);1401
+ p.pu[LUMA_24x32].luma_vss = PFX(interp_8tap_vert_ss_24x32_avx512);1402
+ p.pu[LUMA_32x8].luma_vss = PFX(interp_8tap_vert_ss_32x8_avx512);1403
+ p.pu[LUMA_32x16].luma_vss = PFX(interp_8tap_vert_ss_32x16_avx512);1404
+ p.pu[LUMA_32x32].luma_vss = PFX(interp_8tap_vert_ss_32x32_avx512);1405
+ p.pu[LUMA_32x24].luma_vss = PFX(interp_8tap_vert_ss_32x24_avx512);1406
+ p.pu[LUMA_32x64].luma_vss = PFX(interp_8tap_vert_ss_32x64_avx512);1407
+ p.pu[LUMA_64x16].luma_vss = PFX(interp_8tap_vert_ss_64x16_avx512);1408
+ p.pu[LUMA_64x32].luma_vss = PFX(interp_8tap_vert_ss_64x32_avx512);1409
+ p.pu[LUMA_64x48].luma_vss = PFX(interp_8tap_vert_ss_64x48_avx512);1410
+ p.pu[LUMA_64x64].luma_vss = PFX(interp_8tap_vert_ss_64x64_avx512);1411
+ p.pu[LUMA_48x64].luma_vss = PFX(interp_8tap_vert_ss_48x64_avx512);1412
+1413
+ p.pu[LUMA_8x8].luma_vsp = PFX(interp_8tap_vert_sp_8x8_avx512);1414
+ p.pu[LUMA_8x16].luma_vsp = PFX(interp_8tap_vert_sp_8x16_avx512);1415
+ p.pu[LUMA_8x32].luma_vsp = PFX(interp_8tap_vert_sp_8x32_avx512);1416
+ p.pu[LUMA_16x4].luma_vsp = PFX(interp_8tap_vert_sp_16x4_avx512);1417
+ p.pu[LUMA_16x8].luma_vsp = PFX(interp_8tap_vert_sp_16x8_avx512);1418
+ p.pu[LUMA_16x12].luma_vsp = PFX(interp_8tap_vert_sp_16x12_avx512);1419
+ p.pu[LUMA_16x16].luma_vsp = PFX(interp_8tap_vert_sp_16x16_avx512);1420
+ p.pu[LUMA_16x32].luma_vsp = PFX(interp_8tap_vert_sp_16x32_avx512);1421
+ p.pu[LUMA_16x64].luma_vsp = PFX(interp_8tap_vert_sp_16x64_avx512);1422
+ p.pu[LUMA_24x32].luma_vsp = PFX(interp_8tap_vert_sp_24x32_avx512);1423
+ p.pu[LUMA_32x8].luma_vsp = PFX(interp_8tap_vert_sp_32x8_avx512);1424
+ p.pu[LUMA_32x16].luma_vsp = PFX(interp_8tap_vert_sp_32x16_avx512);1425
+ p.pu[LUMA_32x32].luma_vsp = PFX(interp_8tap_vert_sp_32x32_avx512);1426
+ p.pu[LUMA_32x24].luma_vsp = PFX(interp_8tap_vert_sp_32x24_avx512);1427
+ p.pu[LUMA_32x64].luma_vsp = PFX(interp_8tap_vert_sp_32x64_avx512);1428
+ p.pu[LUMA_64x16].luma_vsp = PFX(interp_8tap_vert_sp_64x16_avx512);1429
+ p.pu[LUMA_64x32].luma_vsp = PFX(interp_8tap_vert_sp_64x32_avx512);1430
+ p.pu[LUMA_64x48].luma_vsp = PFX(interp_8tap_vert_sp_64x48_avx512);1431
+ p.pu[LUMA_64x64].luma_vsp = PFX(interp_8tap_vert_sp_64x64_avx512);1432
+ p.pu[LUMA_48x64].luma_vsp = PFX(interp_8tap_vert_sp_48x64_avx512);1433
+1434
+ p.pu[LUMA_16x4].luma_vpp = PFX(interp_8tap_vert_pp_16x4_avx512);1435
+ p.pu[LUMA_16x8].luma_vpp = PFX(interp_8tap_vert_pp_16x8_avx512);1436
+ p.pu[LUMA_16x12].luma_vpp = PFX(interp_8tap_vert_pp_16x12_avx512);1437
+ p.pu[LUMA_16x16].luma_vpp = PFX(interp_8tap_vert_pp_16x16_avx512);1438
+ p.pu[LUMA_16x32].luma_vpp = PFX(interp_8tap_vert_pp_16x32_avx512);1439
+ p.pu[LUMA_16x64].luma_vpp = PFX(interp_8tap_vert_pp_16x64_avx512);1440
+ p.pu[LUMA_24x32].luma_vpp = PFX(interp_8tap_vert_pp_24x32_avx512);1441
+ p.pu[LUMA_32x8].luma_vpp = PFX(interp_8tap_vert_pp_32x8_avx512);1442
+ p.pu[LUMA_32x16].luma_vpp = PFX(interp_8tap_vert_pp_32x16_avx512);1443
+ p.pu[LUMA_32x32].luma_vpp = PFX(interp_8tap_vert_pp_32x32_avx512);1444
+ p.pu[LUMA_32x24].luma_vpp = PFX(interp_8tap_vert_pp_32x24_avx512);1445
+ p.pu[LUMA_32x64].luma_vpp = PFX(interp_8tap_vert_pp_32x64_avx512);1446
+ p.pu[LUMA_48x64].luma_vpp = PFX(interp_8tap_vert_pp_48x64_avx512);1447
+ p.pu[LUMA_64x16].luma_vpp = PFX(interp_8tap_vert_pp_64x16_avx512);1448
+ p.pu[LUMA_64x32].luma_vpp = PFX(interp_8tap_vert_pp_64x32_avx512);1449
+ p.pu[LUMA_64x48].luma_vpp = PFX(interp_8tap_vert_pp_64x48_avx512);1450
+ p.pu[LUMA_64x64].luma_vpp = PFX(interp_8tap_vert_pp_64x64_avx512);1451
+1452
+ p.pu[LUMA_16x4].luma_vps = PFX(interp_8tap_vert_ps_16x4_avx512);1453
+ p.pu[LUMA_16x8].luma_vps = PFX(interp_8tap_vert_ps_16x8_avx512);1454
+ p.pu[LUMA_16x12].luma_vps = PFX(interp_8tap_vert_ps_16x12_avx512);1455
+ p.pu[LUMA_16x16].luma_vps = PFX(interp_8tap_vert_ps_16x16_avx512);1456
+ p.pu[LUMA_16x32].luma_vps = PFX(interp_8tap_vert_ps_16x32_avx512);1457
+ p.pu[LUMA_16x64].luma_vps = PFX(interp_8tap_vert_ps_16x64_avx512);1458
+ p.pu[LUMA_24x32].luma_vps = PFX(interp_8tap_vert_ps_24x32_avx512);1459
+ p.pu[LUMA_32x8].luma_vps = PFX(interp_8tap_vert_ps_32x8_avx512);1460
+ p.pu[LUMA_32x16].luma_vps = PFX(interp_8tap_vert_ps_32x16_avx512);1461
+ p.pu[LUMA_32x32].luma_vps = PFX(interp_8tap_vert_ps_32x32_avx512);1462
+ p.pu[LUMA_32x24].luma_vps = PFX(interp_8tap_vert_ps_32x24_avx512);1463
+ p.pu[LUMA_32x64].luma_vps = PFX(interp_8tap_vert_ps_32x64_avx512);1464
+ p.pu[LUMA_48x64].luma_vps = PFX(interp_8tap_vert_ps_48x64_avx512);1465
+ p.pu[LUMA_64x16].luma_vps = PFX(interp_8tap_vert_ps_64x16_avx512);1466
+ p.pu[LUMA_64x32].luma_vps = PFX(interp_8tap_vert_ps_64x32_avx512);1467
+ p.pu[LUMA_64x48].luma_vps = PFX(interp_8tap_vert_ps_64x48_avx512);1468
+ p.pu[LUMA_64x64].luma_vps = PFX(interp_8tap_vert_ps_64x64_avx512);1469
+1470
+ p.cu[BLOCK_8x8].dct = PFX(dct8_avx512);1471
+ /* TODO: Currently these kernels performance are similar to AVX2 version, we need a to improve them further to ebable1472
+ * it. Probably a Vtune analysis will help here.1473
+1474
+ * p.cu[BLOCK_16x16].dct = PFX(dct16_avx512);1475
+ * p.cu[BLOCK_32x32].dct = PFX(dct32_avx512); */1476
+1477
+ p.cu[BLOCK_8x8].idct = PFX(idct8_avx512);1478
+ p.cu[BLOCK_16x16].idct = PFX(idct16_avx512);1479
+ p.cu[BLOCK_32x32].idct = PFX(idct32_avx512);1480
+ p.quant = PFX(quant_avx512);1481
+ p.nquant = PFX(nquant_avx512);1482
+ p.denoiseDct = PFX(denoise_dct_avx512);1483
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x32].filter_hps = PFX(interp_4tap_horiz_ps_32x32_avx512);1484
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x16].filter_hps = PFX(interp_4tap_horiz_ps_32x16_avx512);1485
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x24].filter_hps = PFX(interp_4tap_horiz_ps_32x24_avx512);1486
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x8].filter_hps = PFX(interp_4tap_horiz_ps_32x8_avx512);1487
+1488
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x64].filter_hps = PFX(interp_4tap_horiz_ps_32x64_avx512);1489
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x32].filter_hps = PFX(interp_4tap_horiz_ps_32x32_avx512);1490
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x48].filter_hps = PFX(interp_4tap_horiz_ps_32x48_avx512);1491
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].filter_hps = PFX(interp_4tap_horiz_ps_32x16_avx512);1492
+1493
+ p.chroma[X265_CSP_I444].pu[LUMA_32x32].filter_hps = PFX(interp_4tap_horiz_ps_32x32_avx512);1494
+ p.chroma[X265_CSP_I444].pu[LUMA_32x16].filter_hps = PFX(interp_4tap_horiz_ps_32x16_avx512);1495
+ p.chroma[X265_CSP_I444].pu[LUMA_32x64].filter_hps = PFX(interp_4tap_horiz_ps_32x64_avx512);1496
+ p.chroma[X265_CSP_I444].pu[LUMA_32x24].filter_hps = PFX(interp_4tap_horiz_ps_32x24_avx512);1497
+ p.chroma[X265_CSP_I444].pu[LUMA_32x8].filter_hps = PFX(interp_4tap_horiz_ps_32x8_avx512);1498
+1499
+ p.chroma[X265_CSP_I444].pu[LUMA_64x64].filter_hps = PFX(interp_4tap_horiz_ps_64x64_avx512);1500
+ p.chroma[X265_CSP_I444].pu[LUMA_64x48].filter_hps = PFX(interp_4tap_horiz_ps_64x48_avx512);1501
+ p.chroma[X265_CSP_I444].pu[LUMA_64x32].filter_hps = PFX(interp_4tap_horiz_ps_64x32_avx512);1502
+ p.chroma[X265_CSP_I444].pu[LUMA_64x16].filter_hps = PFX(interp_4tap_horiz_ps_64x16_avx512);1503
+1504
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x16].filter_hps = PFX(interp_4tap_horiz_ps_16x16_avx512);1505
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x8].filter_hps = PFX(interp_4tap_horiz_ps_16x8_avx512);1506
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x32].filter_hps = PFX(interp_4tap_horiz_ps_16x32_avx512);1507
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x12].filter_hps = PFX(interp_4tap_horiz_ps_16x12_avx512);1508
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x4].filter_hps = PFX(interp_4tap_horiz_ps_16x4_avx512);1509
+1510
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x32].filter_hps = PFX(interp_4tap_horiz_ps_16x32_avx512);1511
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x16].filter_hps = PFX(interp_4tap_horiz_ps_16x16_avx512);1512
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x64].filter_hps = PFX(interp_4tap_horiz_ps_16x64_avx512);1513
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x24].filter_hps = PFX(interp_4tap_horiz_ps_16x24_avx512);1514
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x8].filter_hps = PFX(interp_4tap_horiz_ps_16x8_avx512);1515
+1516
+ p.chroma[X265_CSP_I444].pu[LUMA_16x16].filter_hps = PFX(interp_4tap_horiz_ps_16x16_avx512);1517
+ p.chroma[X265_CSP_I444].pu[LUMA_16x8].filter_hps = PFX(interp_4tap_horiz_ps_16x8_avx512);1518
+ p.chroma[X265_CSP_I444].pu[LUMA_16x32].filter_hps = PFX(interp_4tap_horiz_ps_16x32_avx512);1519
+ p.chroma[X265_CSP_I444].pu[LUMA_16x12].filter_hps = PFX(interp_4tap_horiz_ps_16x12_avx512);1520
+ p.chroma[X265_CSP_I444].pu[LUMA_16x4].filter_hps = PFX(interp_4tap_horiz_ps_16x4_avx512);1521
+ p.chroma[X265_CSP_I444].pu[LUMA_16x64].filter_hps = PFX(interp_4tap_horiz_ps_16x64_avx512);1522
+1523
+ p.chroma[X265_CSP_I444].pu[LUMA_48x64].filter_hps = PFX(interp_4tap_horiz_ps_48x64_avx512);1524
+1525
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_8x8].filter_hps = PFX(interp_4tap_horiz_ps_8x8_avx512);1526
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_8x4].filter_hps = PFX(interp_4tap_horiz_ps_8x4_avx512);1527
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_8x16].filter_hps = PFX(interp_4tap_horiz_ps_8x16_avx512);1528
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_8x32].filter_hps = PFX(interp_4tap_horiz_ps_8x32_avx512);1529
+1530
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x8].filter_hps = PFX(interp_4tap_horiz_ps_8x8_avx512);1531
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x16].filter_hps = PFX(interp_4tap_horiz_ps_8x16_avx512);1532
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x32].filter_hps = PFX(interp_4tap_horiz_ps_8x32_avx512);1533
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x12].filter_hps = PFX(interp_4tap_horiz_ps_8x12_avx512);1534
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x64].filter_hps = PFX(interp_4tap_horiz_ps_8x64_avx512);1535
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x4].filter_hps = PFX(interp_4tap_horiz_ps_8x4_avx512);1536
+1537
+ p.chroma[X265_CSP_I444].pu[LUMA_8x8].filter_hps = PFX(interp_4tap_horiz_ps_8x8_avx512);1538
+ p.chroma[X265_CSP_I444].pu[LUMA_8x4].filter_hps = PFX(interp_4tap_horiz_ps_8x4_avx512);1539
+ p.chroma[X265_CSP_I444].pu[LUMA_8x16].filter_hps = PFX(interp_4tap_horiz_ps_8x16_avx512);1540
+ p.chroma[X265_CSP_I444].pu[LUMA_8x32].filter_hps = PFX(interp_4tap_horiz_ps_8x32_avx512);1541
+1542
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_24x32].filter_hps = PFX(interp_4tap_horiz_ps_24x32_avx512);1543
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_24x64].filter_hps = PFX(interp_4tap_horiz_ps_24x64_avx512);1544
+ p.chroma[X265_CSP_I444].pu[LUMA_24x32].filter_hps = PFX(interp_4tap_horiz_ps_24x32_avx512);1545
+1546
+ //Luma_hps_32xN1547
+ p.pu[LUMA_32x8].luma_hps = PFX(interp_8tap_horiz_ps_32x8_avx512);1548
+ p.pu[LUMA_32x16].luma_hps = PFX(interp_8tap_horiz_ps_32x16_avx512);1549
+ p.pu[LUMA_32x32].luma_hps = PFX(interp_8tap_horiz_ps_32x32_avx512);1550
+ p.pu[LUMA_32x24].luma_hps = PFX(interp_8tap_horiz_ps_32x24_avx512);1551
+ p.pu[LUMA_32x64].luma_hps = PFX(interp_8tap_horiz_ps_32x64_avx512);1552
+ //Luma_hps_64xN1553
+ p.pu[LUMA_64x16].luma_hps = PFX(interp_8tap_horiz_ps_64x16_avx512);1554
+ p.pu[LUMA_64x32].luma_hps = PFX(interp_8tap_horiz_ps_64x32_avx512);1555
+ p.pu[LUMA_64x48].luma_hps = PFX(interp_8tap_horiz_ps_64x48_avx512);1556
+ p.pu[LUMA_64x64].luma_hps = PFX(interp_8tap_horiz_ps_64x64_avx512);1557
+ //Luma_hps_16xN1558
+ p.pu[LUMA_16x4].luma_hps = PFX(interp_8tap_horiz_ps_16x4_avx512);1559
+ p.pu[LUMA_16x8].luma_hps = PFX(interp_8tap_horiz_ps_16x8_avx512);1560
+ p.pu[LUMA_16x12].luma_hps = PFX(interp_8tap_horiz_ps_16x12_avx512);1561
+ p.pu[LUMA_16x16].luma_hps = PFX(interp_8tap_horiz_ps_16x16_avx512);1562
+ p.pu[LUMA_16x32].luma_hps = PFX(interp_8tap_horiz_ps_16x32_avx512);1563
+ p.pu[LUMA_16x64].luma_hps = PFX(interp_8tap_horiz_ps_16x64_avx512);1564
+ //Luma_hps_48x641565
+ p.pu[LUMA_48x64].luma_hps = PFX(interp_8tap_horiz_ps_48x64_avx512);1566
+ //Luma_hps_24x321567
+ p.pu[LUMA_24x32].luma_hps = PFX(interp_8tap_horiz_ps_24x32_avx512);1568
+ //Luma_hps_8xN1569
+ p.pu[LUMA_8x4].luma_hps = PFX(interp_8tap_horiz_ps_8x4_avx512);1570
+ p.pu[LUMA_8x8].luma_hps = PFX(interp_8tap_horiz_ps_8x8_avx512);1571
+ p.pu[LUMA_8x16].luma_hps = PFX(interp_8tap_horiz_ps_8x16_avx512);1572
+ p.pu[LUMA_8x32].luma_hps = PFX(interp_8tap_horiz_ps_8x32_avx512);1573
+ p.pu[LUMA_16x8].satd = PFX(pixel_satd_16x8_avx512);1574
+ p.pu[LUMA_16x16].satd = PFX(pixel_satd_16x16_avx512);1575
+ p.pu[LUMA_16x32].satd = PFX(pixel_satd_16x32_avx512);1576
+ p.pu[LUMA_16x64].satd = PFX(pixel_satd_16x64_avx512);1577
+ p.pu[LUMA_32x8].satd = PFX(pixel_satd_32x8_avx512);1578
+ p.pu[LUMA_32x16].satd = PFX(pixel_satd_32x16_avx512);1579
+ p.pu[LUMA_32x24].satd = PFX(pixel_satd_32x24_avx512);1580
+ p.pu[LUMA_32x32].satd = PFX(pixel_satd_32x32_avx512);1581
+ p.pu[LUMA_32x64].satd = PFX(pixel_satd_32x64_avx512);1582
+ p.pu[LUMA_64x16].satd = PFX(pixel_satd_64x16_avx512);1583
+ p.pu[LUMA_64x32].satd = PFX(pixel_satd_64x32_avx512);1584
+ p.pu[LUMA_64x48].satd = PFX(pixel_satd_64x48_avx512);1585
+ p.pu[LUMA_64x64].satd = PFX(pixel_satd_64x64_avx512);1586
+ p.pu[LUMA_48x64].satd = PFX(pixel_satd_48x64_avx512);1587
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x32].satd = PFX(pixel_satd_16x32_avx512);1588
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x16].satd = PFX(pixel_satd_16x16_avx512);1589
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x8].satd = PFX(pixel_satd_16x8_avx512);1590
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x32].satd = PFX(pixel_satd_32x32_avx512);1591
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x16].satd = PFX(pixel_satd_32x16_avx512);1592
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x24].satd = PFX(pixel_satd_32x24_avx512);1593
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x8].satd = PFX(pixel_satd_32x8_avx512);1594
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x64].satd = PFX(pixel_satd_16x64_avx512);1595
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x32].satd = PFX(pixel_satd_16x32_avx512);1596
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x16].satd = PFX(pixel_satd_16x16_avx512);1597
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x8].satd = PFX(pixel_satd_16x8_avx512);1598
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x64].satd = PFX(pixel_satd_32x64_avx512);1599
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x32].satd = PFX(pixel_satd_32x32_avx512);1600
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].satd = PFX(pixel_satd_32x16_avx512);1601
+1602
+ p.cu[BLOCK_32x32].intra_pred[DC_IDX] = PFX(intra_pred_dc32_avx512);1603
+ p.cu[BLOCK_32x32].intra_pred[2] = PFX(intra_pred_ang32_2_avx512);1604
+ p.cu[BLOCK_32x32].intra_pred[34] = PFX(intra_pred_ang32_2_avx512);1605
+ p.cu[BLOCK_32x32].intra_pred[9] = PFX(intra_pred_ang32_9_avx512);1606
+ p.cu[BLOCK_32x32].intra_pred[10] = PFX(intra_pred_ang32_10_avx512);1607
+ p.cu[BLOCK_32x32].intra_pred[11] = PFX(intra_pred_ang32_11_avx512);1608
+ p.cu[BLOCK_32x32].intra_pred[18] = PFX(intra_pred_ang32_18_avx512);1609
+ p.cu[BLOCK_32x32].intra_pred[25] = PFX(intra_pred_ang32_25_avx512);1610
+ p.cu[BLOCK_32x32].intra_pred[26] = PFX(intra_pred_ang32_26_avx512);1611
+ p.cu[BLOCK_32x32].intra_pred[27] = PFX(intra_pred_ang32_27_avx512);1612
+ p.cu[BLOCK_32x32].intra_pred[5] = PFX(intra_pred_ang32_5_avx512);1613
+ p.cu[BLOCK_32x32].intra_pred[31] = PFX(intra_pred_ang32_31_avx512);1614
+ p.cu[BLOCK_32x32].intra_pred[32] = PFX(intra_pred_ang32_32_avx512);1615
+ p.cu[BLOCK_32x32].intra_pred[4] = PFX(intra_pred_ang32_4_avx512);1616
+ p.cu[BLOCK_32x32].intra_pred[30] = PFX(intra_pred_ang32_30_avx512);1617
+ p.cu[BLOCK_32x32].intra_pred[6] = PFX(intra_pred_ang32_6_avx512);1618
+ p.cu[BLOCK_32x32].intra_pred[29] = PFX(intra_pred_ang32_29_avx512);1619
+ p.cu[BLOCK_32x32].intra_pred[7] = PFX(intra_pred_ang32_7_avx512);1620
+ p.cu[BLOCK_32x32].intra_pred[8] = PFX(intra_pred_ang32_8_avx512);1621
+ p.cu[BLOCK_32x32].intra_pred[28] = PFX(intra_pred_ang32_28_avx512);1622
+ p.cu[BLOCK_16x16].intra_pred[9] = PFX(intra_pred_ang16_9_avx512);1623
+ p.cu[BLOCK_16x16].intra_pred[11] = PFX(intra_pred_ang16_11_avx512);1624
+ p.cu[BLOCK_16x16].intra_pred[25] = PFX(intra_pred_ang16_25_avx512);1625
+ p.cu[BLOCK_16x16].intra_pred[27] = PFX(intra_pred_ang16_27_avx512);1626
+ p.cu[BLOCK_16x16].intra_pred[8] = PFX(intra_pred_ang16_8_avx512);1627
+ p.cu[BLOCK_16x16].intra_pred[28] = PFX(intra_pred_ang16_28_avx512);1628
+ p.cu[BLOCK_16x16].intra_pred[5] = PFX(intra_pred_ang16_5_avx512);1629
+ p.cu[BLOCK_16x16].intra_pred[31] = PFX(intra_pred_ang16_31_avx512);1630
+ p.cu[BLOCK_16x16].intra_pred[4] = PFX(intra_pred_ang16_4_avx512);1631
+ p.cu[BLOCK_16x16].intra_pred[32] = PFX(intra_pred_ang16_32_avx512);1632
+ p.cu[BLOCK_16x16].intra_pred[6] = PFX(intra_pred_ang16_6_avx512);1633
+ p.cu[BLOCK_16x16].intra_pred[30] = PFX(intra_pred_ang16_30_avx512);1634
+ p.cu[BLOCK_16x16].intra_pred[7] = PFX(intra_pred_ang16_7_avx512);1635
+ p.cu[BLOCK_16x16].intra_pred[29] = PFX(intra_pred_ang16_29_avx512);1636
+ p.pu[LUMA_64x64].luma_hvpp = interp_8tap_hv_pp_cpu<LUMA_64x64>;1637
+ p.pu[LUMA_64x48].luma_hvpp = interp_8tap_hv_pp_cpu<LUMA_64x48>;1638
+ p.pu[LUMA_64x32].luma_hvpp = interp_8tap_hv_pp_cpu<LUMA_64x32>;1639
+ p.pu[LUMA_64x16].luma_hvpp = interp_8tap_hv_pp_cpu<LUMA_64x16>;1640
+ p.pu[LUMA_32x8].luma_hvpp = interp_8tap_hv_pp_cpu<LUMA_32x8>;1641
+ p.pu[LUMA_32x16].luma_hvpp = interp_8tap_hv_pp_cpu<LUMA_32x16>;1642
+ p.pu[LUMA_32x32].luma_hvpp = interp_8tap_hv_pp_cpu<LUMA_32x32>;1643
+ p.pu[LUMA_32x24].luma_hvpp = interp_8tap_hv_pp_cpu<LUMA_32x24>;1644
+ p.pu[LUMA_32x64].luma_hvpp = interp_8tap_hv_pp_cpu<LUMA_32x64>;1645
+ p.pu[LUMA_16x4].luma_hvpp = interp_8tap_hv_pp_cpu<LUMA_16x4>;1646
+ p.pu[LUMA_16x8].luma_hvpp = interp_8tap_hv_pp_cpu<LUMA_16x8>;1647
+ p.pu[LUMA_16x12].luma_hvpp = interp_8tap_hv_pp_cpu<LUMA_16x12>;1648
+ p.pu[LUMA_16x16].luma_hvpp = interp_8tap_hv_pp_cpu<LUMA_16x16>;1649
+ p.pu[LUMA_16x32].luma_hvpp = interp_8tap_hv_pp_cpu<LUMA_16x32>;1650
+ p.pu[LUMA_16x64].luma_hvpp = interp_8tap_hv_pp_cpu<LUMA_16x64>;1651
+ p.pu[LUMA_48x64].luma_hvpp = interp_8tap_hv_pp_cpu<LUMA_48x64>;1652
+1653
+ p.cu[BLOCK_16x16].psy_cost_pp = PFX(psyCost_pp_16x16_avx512);1654
+ p.cu[BLOCK_32x32].psy_cost_pp = PFX(psyCost_pp_32x32_avx512);1655
+ p.cu[BLOCK_64x64].psy_cost_pp = PFX(psyCost_pp_64x64_avx512);1656
+1657
+ p.cu[BLOCK_4x4].nonPsyRdoQuant = PFX(nonPsyRdoQuant4_avx512);1658
+ p.cu[BLOCK_8x8].nonPsyRdoQuant = PFX(nonPsyRdoQuant8_avx512);1659
+ p.cu[BLOCK_16x16].nonPsyRdoQuant = PFX(nonPsyRdoQuant16_avx512);1660
+ p.cu[BLOCK_32x32].nonPsyRdoQuant = PFX(nonPsyRdoQuant32_avx512);1661
+ p.cu[BLOCK_4x4].psyRdoQuant = PFX(psyRdoQuant4_avx512);1662
+ p.cu[BLOCK_8x8].psyRdoQuant = PFX(psyRdoQuant8_avx512);1663
+ p.cu[BLOCK_16x16].psyRdoQuant = PFX(psyRdoQuant16_avx512);1664
+ p.cu[BLOCK_32x32].psyRdoQuant = PFX(psyRdoQuant32_avx512);1665
+1666
+ p.cu[BLOCK_32x32].sse_ss = (pixel_sse_ss_t)PFX(pixel_ssd_32x32_avx512);1667
+ p.cu[BLOCK_64x64].sse_ss = (pixel_sse_ss_t)PFX(pixel_ssd_64x64_avx512);1668
+ p.cu[BLOCK_32x32].sse_pp = PFX(pixel_ssd_32x32_avx512);1669
+ p.cu[BLOCK_64x64].sse_pp = PFX(pixel_ssd_64x64_avx512);1670
+ p.chroma[X265_CSP_I420].cu[BLOCK_420_32x32].sse_pp = (pixel_sse_t)PFX(pixel_ssd_32x32_avx512);1671
+ p.chroma[X265_CSP_I422].cu[BLOCK_422_32x64].sse_pp = (pixel_sse_t)PFX(pixel_ssd_32x64_avx512);1672
+ p.planecopy_sp_shl = PFX(upShift_16_avx512);1673
+1674
+ }1675
+#endif1676
}1677
#else // if HIGH_BIT_DEPTH1678
1679
1680
//p.frameInitLowres = PFX(frame_init_lowres_core_mmx2);1681
p.frameInitLowres = PFX(frame_init_lowres_core_sse2);1682
1683
- ALL_LUMA_TU(blockfill_s, blockfill_s, sse2);1684
+ ALL_LUMA_TU(blockfill_s[NONALIGNED], blockfill_s, sse2);1685
+ ALL_LUMA_TU(blockfill_s[ALIGNED], blockfill_s, sse2);1686
ALL_LUMA_TU_S(cpy2Dto1D_shl, cpy2Dto1D_shl_, sse2);1687
ALL_LUMA_TU_S(cpy2Dto1D_shr, cpy2Dto1D_shr_, sse2);1688
- ALL_LUMA_TU_S(cpy1Dto2D_shl, cpy1Dto2D_shl_, sse2);1689
+ ALL_LUMA_TU_S(cpy1Dto2D_shl[ALIGNED], cpy1Dto2D_shl_, sse2);1690
+ ALL_LUMA_TU_S(cpy1Dto2D_shl[NONALIGNED], cpy1Dto2D_shl_, sse2);1691
ALL_LUMA_TU_S(cpy1Dto2D_shr, cpy1Dto2D_shr_, sse2);1692
- ALL_LUMA_TU_S(ssd_s, pixel_ssd_s_, sse2);1693
-1694
+ ALL_LUMA_TU_S(ssd_s[NONALIGNED], pixel_ssd_s_, sse2);1695
ALL_LUMA_TU_S(intra_pred[PLANAR_IDX], intra_pred_planar, sse2);1696
ALL_LUMA_TU_S(intra_pred[DC_IDX], intra_pred_dc, sse2);1697
-1698
p.cu[BLOCK_4x4].intra_pred[2] = PFX(intra_pred_ang4_2_sse2);1699
p.cu[BLOCK_4x4].intra_pred[3] = PFX(intra_pred_ang4_3_sse2);1700
p.cu[BLOCK_4x4].intra_pred[4] = PFX(intra_pred_ang4_4_sse2);1701
1702
p.cu[BLOCK_4x4].intra_pred[33] = PFX(intra_pred_ang4_33_sse2);1703
1704
p.cu[BLOCK_4x4].intra_pred_allangs = PFX(all_angs_pred_4x4_sse2);1705
-1706
- p.cu[BLOCK_4x4].calcresidual = PFX(getResidual4_sse2);1707
- p.cu[BLOCK_8x8].calcresidual = PFX(getResidual8_sse2);1708
+ ASSIGN2(p.cu[BLOCK_4x4].calcresidual, getResidual4_sse2);1709
+ ASSIGN2(p.cu[BLOCK_8x8].calcresidual, getResidual8_sse2);1710
1711
ALL_LUMA_TU_S(transpose, transpose, sse2);1712
p.cu[BLOCK_64x64].transpose = PFX(transpose64_sse2);1713
1714
p.dst4x4 = PFX(dst4_sse2);1715
1716
p.planecopy_sp = PFX(downShift_16_sse2);1717
- ALL_CHROMA_420_PU(p2s, filterPixelToShort, sse2);1718
- ALL_CHROMA_422_PU(p2s, filterPixelToShort, sse2);1719
- ALL_CHROMA_444_PU(p2s, filterPixelToShort, sse2);1720
- ALL_LUMA_PU(convert_p2s, filterPixelToShort, sse2);1721
+ ALL_CHROMA_420_PU(p2s[NONALIGNED], filterPixelToShort, sse2);1722
+ ALL_CHROMA_422_PU(p2s[NONALIGNED], filterPixelToShort, sse2);1723
+ ALL_CHROMA_444_PU(p2s[NONALIGNED], filterPixelToShort, sse2);1724
+ ALL_CHROMA_420_PU(p2s[ALIGNED], filterPixelToShort, sse2);1725
+ ALL_CHROMA_422_PU(p2s[ALIGNED], filterPixelToShort, sse2);1726
+ ALL_CHROMA_444_PU(p2s[ALIGNED], filterPixelToShort, sse2);1727
+ ALL_LUMA_PU(convert_p2s[NONALIGNED], filterPixelToShort, sse2);1728
+ ALL_LUMA_PU(convert_p2s[ALIGNED], filterPixelToShort, sse2);1729
ALL_LUMA_TU(count_nonzero, count_nonzero, sse2);1730
p.propagateCost = PFX(mbtree_propagate_cost_sse2);1731
}1732
1733
p.pu[LUMA_8x8].luma_hvpp = PFX(interp_8tap_hv_pp_8x8_ssse3);1734
1735
p.frameInitLowres = PFX(frame_init_lowres_core_ssse3);1736
- p.scale1D_128to64 = PFX(scale1D_128to64_ssse3);1737
+ ASSIGN2(p.scale1D_128to64, scale1D_128to64_ssse3);1738
p.scale2D_64to32 = PFX(scale2D_64to32_ssse3);1739
1740
- p.pu[LUMA_8x4].convert_p2s = PFX(filterPixelToShort_8x4_ssse3);1741
- p.pu[LUMA_8x8].convert_p2s = PFX(filterPixelToShort_8x8_ssse3);1742
- p.pu[LUMA_8x16].convert_p2s = PFX(filterPixelToShort_8x16_ssse3);1743
- p.pu[LUMA_8x32].convert_p2s = PFX(filterPixelToShort_8x32_ssse3);1744
- p.pu[LUMA_16x4].convert_p2s = PFX(filterPixelToShort_16x4_ssse3);1745
- p.pu[LUMA_16x8].convert_p2s = PFX(filterPixelToShort_16x8_ssse3);1746
- p.pu[LUMA_16x12].convert_p2s = PFX(filterPixelToShort_16x12_ssse3);1747
- p.pu[LUMA_16x16].convert_p2s = PFX(filterPixelToShort_16x16_ssse3);1748
- p.pu[LUMA_16x32].convert_p2s = PFX(filterPixelToShort_16x32_ssse3);1749
- p.pu[LUMA_16x64].convert_p2s = PFX(filterPixelToShort_16x64_ssse3);1750
- p.pu[LUMA_32x8].convert_p2s = PFX(filterPixelToShort_32x8_ssse3);1751
- p.pu[LUMA_32x16].convert_p2s = PFX(filterPixelToShort_32x16_ssse3);1752
- p.pu[LUMA_32x24].convert_p2s = PFX(filterPixelToShort_32x24_ssse3);1753
- p.pu[LUMA_32x32].convert_p2s = PFX(filterPixelToShort_32x32_ssse3);1754
- p.pu[LUMA_32x64].convert_p2s = PFX(filterPixelToShort_32x64_ssse3);1755
- p.pu[LUMA_64x16].convert_p2s = PFX(filterPixelToShort_64x16_ssse3);1756
- p.pu[LUMA_64x32].convert_p2s = PFX(filterPixelToShort_64x32_ssse3);1757
- p.pu[LUMA_64x48].convert_p2s = PFX(filterPixelToShort_64x48_ssse3);1758
- p.pu[LUMA_64x64].convert_p2s = PFX(filterPixelToShort_64x64_ssse3);1759
- p.pu[LUMA_12x16].convert_p2s = PFX(filterPixelToShort_12x16_ssse3);1760
- p.pu[LUMA_24x32].convert_p2s = PFX(filterPixelToShort_24x32_ssse3);1761
- p.pu[LUMA_48x64].convert_p2s = PFX(filterPixelToShort_48x64_ssse3);1762
-1763
- p.chroma[X265_CSP_I420].pu[CHROMA_420_8x2].p2s = PFX(filterPixelToShort_8x2_ssse3);1764
- p.chroma[X265_CSP_I420].pu[CHROMA_420_8x4].p2s = PFX(filterPixelToShort_8x4_ssse3);1765
- p.chroma[X265_CSP_I420].pu[CHROMA_420_8x6].p2s = PFX(filterPixelToShort_8x6_ssse3);1766
- p.chroma[X265_CSP_I420].pu[CHROMA_420_8x8].p2s = PFX(filterPixelToShort_8x8_ssse3);1767
- p.chroma[X265_CSP_I420].pu[CHROMA_420_8x16].p2s = PFX(filterPixelToShort_8x16_ssse3);1768
- p.chroma[X265_CSP_I420].pu[CHROMA_420_8x32].p2s = PFX(filterPixelToShort_8x32_ssse3);1769
- p.chroma[X265_CSP_I420].pu[CHROMA_420_16x4].p2s = PFX(filterPixelToShort_16x4_ssse3);1770
- p.chroma[X265_CSP_I420].pu[CHROMA_420_16x8].p2s = PFX(filterPixelToShort_16x8_ssse3);1771
- p.chroma[X265_CSP_I420].pu[CHROMA_420_16x12].p2s = PFX(filterPixelToShort_16x12_ssse3);1772
- p.chroma[X265_CSP_I420].pu[CHROMA_420_16x16].p2s = PFX(filterPixelToShort_16x16_ssse3);1773
- p.chroma[X265_CSP_I420].pu[CHROMA_420_16x32].p2s = PFX(filterPixelToShort_16x32_ssse3);1774
- p.chroma[X265_CSP_I420].pu[CHROMA_420_32x8].p2s = PFX(filterPixelToShort_32x8_ssse3);1775
- p.chroma[X265_CSP_I420].pu[CHROMA_420_32x16].p2s = PFX(filterPixelToShort_32x16_ssse3);1776
- p.chroma[X265_CSP_I420].pu[CHROMA_420_32x24].p2s = PFX(filterPixelToShort_32x24_ssse3);1777
- p.chroma[X265_CSP_I420].pu[CHROMA_420_32x32].p2s = PFX(filterPixelToShort_32x32_ssse3);1778
- p.chroma[X265_CSP_I422].pu[CHROMA_422_8x4].p2s = PFX(filterPixelToShort_8x4_ssse3);1779
- p.chroma[X265_CSP_I422].pu[CHROMA_422_8x8].p2s = PFX(filterPixelToShort_8x8_ssse3);1780
- p.chroma[X265_CSP_I422].pu[CHROMA_422_8x12].p2s = PFX(filterPixelToShort_8x12_ssse3);1781
- p.chroma[X265_CSP_I422].pu[CHROMA_422_8x16].p2s = PFX(filterPixelToShort_8x16_ssse3);1782
- p.chroma[X265_CSP_I422].pu[CHROMA_422_8x32].p2s = PFX(filterPixelToShort_8x32_ssse3);1783
- p.chroma[X265_CSP_I422].pu[CHROMA_422_8x64].p2s = PFX(filterPixelToShort_8x64_ssse3);1784
- p.chroma[X265_CSP_I422].pu[CHROMA_422_12x32].p2s = PFX(filterPixelToShort_12x32_ssse3);1785
- p.chroma[X265_CSP_I422].pu[CHROMA_422_16x8].p2s = PFX(filterPixelToShort_16x8_ssse3);1786
- p.chroma[X265_CSP_I422].pu[CHROMA_422_16x16].p2s = PFX(filterPixelToShort_16x16_ssse3);1787
- p.chroma[X265_CSP_I422].pu[CHROMA_422_16x24].p2s = PFX(filterPixelToShort_16x24_ssse3);1788
- p.chroma[X265_CSP_I422].pu[CHROMA_422_16x32].p2s = PFX(filterPixelToShort_16x32_ssse3);1789
- p.chroma[X265_CSP_I422].pu[CHROMA_422_16x64].p2s = PFX(filterPixelToShort_16x64_ssse3);1790
- p.chroma[X265_CSP_I422].pu[CHROMA_422_24x64].p2s = PFX(filterPixelToShort_24x64_ssse3);1791
- p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].p2s = PFX(filterPixelToShort_32x16_ssse3);1792
- p.chroma[X265_CSP_I422].pu[CHROMA_422_32x32].p2s = PFX(filterPixelToShort_32x32_ssse3);1793
- p.chroma[X265_CSP_I422].pu[CHROMA_422_32x48].p2s = PFX(filterPixelToShort_32x48_ssse3);1794
- p.chroma[X265_CSP_I422].pu[CHROMA_422_32x64].p2s = PFX(filterPixelToShort_32x64_ssse3);1795
+ ASSIGN2(p.pu[LUMA_8x4].convert_p2s, filterPixelToShort_8x4_ssse3);1796
+ ASSIGN2(p.pu[LUMA_8x8].convert_p2s, filterPixelToShort_8x8_ssse3);1797
+ ASSIGN2(p.pu[LUMA_8x16].convert_p2s, filterPixelToShort_8x16_ssse3);1798
+ ASSIGN2(p.pu[LUMA_8x32].convert_p2s, filterPixelToShort_8x32_ssse3);1799
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_8x2].p2s, filterPixelToShort_8x2_ssse3);1800
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_8x4].p2s, filterPixelToShort_8x4_ssse3);1801
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_8x6].p2s, filterPixelToShort_8x6_ssse3);1802
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_8x8].p2s, filterPixelToShort_8x8_ssse3);1803
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_8x16].p2s, filterPixelToShort_8x16_ssse3);1804
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_8x32].p2s, filterPixelToShort_8x32_ssse3);1805
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_8x4].p2s, filterPixelToShort_8x4_ssse3);1806
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_8x8].p2s, filterPixelToShort_8x8_ssse3);1807
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_8x12].p2s, filterPixelToShort_8x12_ssse3);1808
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_8x16].p2s, filterPixelToShort_8x16_ssse3);1809
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_8x32].p2s, filterPixelToShort_8x32_ssse3);1810
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_8x64].p2s, filterPixelToShort_8x64_ssse3);1811
+1812
+ ASSIGN2(p.pu[LUMA_16x4].convert_p2s, filterPixelToShort_16x4_ssse3);1813
+ ASSIGN2(p.pu[LUMA_16x8].convert_p2s, filterPixelToShort_16x8_ssse3);1814
+ ASSIGN2(p.pu[LUMA_16x12].convert_p2s, filterPixelToShort_16x12_ssse3);1815
+ ASSIGN2(p.pu[LUMA_16x16].convert_p2s, filterPixelToShort_16x16_ssse3);1816
+ ASSIGN2(p.pu[LUMA_16x32].convert_p2s, filterPixelToShort_16x32_ssse3);1817
+ ASSIGN2(p.pu[LUMA_16x64].convert_p2s, filterPixelToShort_16x64_ssse3);1818
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_16x4].p2s, filterPixelToShort_16x4_ssse3);1819
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_16x8].p2s, filterPixelToShort_16x8_ssse3);1820
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_16x12].p2s, filterPixelToShort_16x12_ssse3);1821
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_16x16].p2s, filterPixelToShort_16x16_ssse3);1822
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_16x32].p2s, filterPixelToShort_16x32_ssse3);1823
+1824
+ ASSIGN2(p.pu[LUMA_32x8].convert_p2s, filterPixelToShort_32x8_ssse3);1825
+ ASSIGN2(p.pu[LUMA_32x16].convert_p2s, filterPixelToShort_32x16_ssse3);1826
+ ASSIGN2(p.pu[LUMA_32x24].convert_p2s, filterPixelToShort_32x24_ssse3);1827
+ ASSIGN2(p.pu[LUMA_32x32].convert_p2s, filterPixelToShort_32x32_ssse3);1828
+ ASSIGN2(p.pu[LUMA_32x64].convert_p2s, filterPixelToShort_32x64_ssse3);1829
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_32x8].p2s, filterPixelToShort_32x8_ssse3);1830
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_32x16].p2s, filterPixelToShort_32x16_ssse3);1831
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_32x24].p2s, filterPixelToShort_32x24_ssse3);1832
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_32x32].p2s, filterPixelToShort_32x32_ssse3);1833
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].p2s, filterPixelToShort_32x16_ssse3);1834
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_32x32].p2s, filterPixelToShort_32x32_ssse3);1835
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_32x48].p2s, filterPixelToShort_32x48_ssse3);1836
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_32x64].p2s, filterPixelToShort_32x64_ssse3);1837
+1838
+ ASSIGN2(p.pu[LUMA_64x16].convert_p2s, filterPixelToShort_64x16_ssse3);1839
+ ASSIGN2(p.pu[LUMA_64x32].convert_p2s, filterPixelToShort_64x32_ssse3);1840
+ ASSIGN2(p.pu[LUMA_64x48].convert_p2s, filterPixelToShort_64x48_ssse3);1841
+ ASSIGN2(p.pu[LUMA_64x64].convert_p2s, filterPixelToShort_64x64_ssse3);1842
+ ASSIGN2(p.pu[LUMA_12x16].convert_p2s, filterPixelToShort_12x16_ssse3);1843
+ ASSIGN2(p.pu[LUMA_24x32].convert_p2s, filterPixelToShort_24x32_ssse3);1844
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_24x64].p2s, filterPixelToShort_24x64_ssse3);1845
+ ASSIGN2(p.pu[LUMA_48x64].convert_p2s, filterPixelToShort_48x64_ssse3);1846
+1847
p.findPosFirstLast = PFX(findPosFirstLast_ssse3);1848
p.fix8Unpack = PFX(cutree_fix8_unpack_ssse3);1849
p.fix8Pack = PFX(cutree_fix8_pack_ssse3);1850
1851
CHROMA_420_CU_BLOCKCOPY(ps, sse4);1852
CHROMA_422_CU_BLOCKCOPY(ps, sse4);1853
1854
- p.cu[BLOCK_16x16].calcresidual = PFX(getResidual16_sse4);1855
- p.cu[BLOCK_32x32].calcresidual = PFX(getResidual32_sse4);1856
+ ASSIGN2(p.cu[BLOCK_16x16].calcresidual, getResidual16_sse4);1857
+ ASSIGN2(p.cu[BLOCK_32x32].calcresidual, getResidual32_sse4);1858
p.cu[BLOCK_8x8].dct = PFX(dct8_sse4);1859
p.denoiseDct = PFX(denoise_dct_sse4);1860
p.quant = PFX(quant_sse4);1861
1862
1863
p.cu[BLOCK_4x4].psy_cost_pp = PFX(psyCost_pp_4x4_sse4);1864
1865
- p.pu[LUMA_4x4].convert_p2s = PFX(filterPixelToShort_4x4_sse4);1866
- p.pu[LUMA_4x8].convert_p2s = PFX(filterPixelToShort_4x8_sse4);1867
- p.pu[LUMA_4x16].convert_p2s = PFX(filterPixelToShort_4x16_sse4);1868
-1869
- p.chroma[X265_CSP_I420].pu[CHROMA_420_2x4].p2s = PFX(filterPixelToShort_2x4_sse4);1870
- p.chroma[X265_CSP_I420].pu[CHROMA_420_2x8].p2s = PFX(filterPixelToShort_2x8_sse4);1871
- p.chroma[X265_CSP_I420].pu[CHROMA_420_4x2].p2s = PFX(filterPixelToShort_4x2_sse4);1872
- p.chroma[X265_CSP_I420].pu[CHROMA_420_4x4].p2s = PFX(filterPixelToShort_4x4_sse4);1873
- p.chroma[X265_CSP_I420].pu[CHROMA_420_4x8].p2s = PFX(filterPixelToShort_4x8_sse4);1874
- p.chroma[X265_CSP_I420].pu[CHROMA_420_4x16].p2s = PFX(filterPixelToShort_4x16_sse4);1875
- p.chroma[X265_CSP_I420].pu[CHROMA_420_6x8].p2s = PFX(filterPixelToShort_6x8_sse4);1876
- p.chroma[X265_CSP_I422].pu[CHROMA_422_2x8].p2s = PFX(filterPixelToShort_2x8_sse4);1877
- p.chroma[X265_CSP_I422].pu[CHROMA_422_2x16].p2s = PFX(filterPixelToShort_2x16_sse4);1878
- p.chroma[X265_CSP_I422].pu[CHROMA_422_4x4].p2s = PFX(filterPixelToShort_4x4_sse4);1879
- p.chroma[X265_CSP_I422].pu[CHROMA_422_4x8].p2s = PFX(filterPixelToShort_4x8_sse4);1880
- p.chroma[X265_CSP_I422].pu[CHROMA_422_4x16].p2s = PFX(filterPixelToShort_4x16_sse4);1881
- p.chroma[X265_CSP_I422].pu[CHROMA_422_4x32].p2s = PFX(filterPixelToShort_4x32_sse4);1882
- p.chroma[X265_CSP_I422].pu[CHROMA_422_6x16].p2s = PFX(filterPixelToShort_6x16_sse4);1883
+ ASSIGN2(p.pu[LUMA_4x4].convert_p2s, filterPixelToShort_4x4_sse4);1884
+ ASSIGN2(p.pu[LUMA_4x8].convert_p2s, filterPixelToShort_4x8_sse4);1885
+ ASSIGN2(p.pu[LUMA_4x16].convert_p2s, filterPixelToShort_4x16_sse4);1886
+1887
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_2x4].p2s, filterPixelToShort_2x4_sse4);1888
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_2x8].p2s, filterPixelToShort_2x8_sse4);1889
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_4x2].p2s, filterPixelToShort_4x2_sse4);1890
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_4x4].p2s, filterPixelToShort_4x4_sse4);1891
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_4x8].p2s, filterPixelToShort_4x8_sse4);1892
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_4x16].p2s, filterPixelToShort_4x16_sse4);1893
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_6x8].p2s, filterPixelToShort_6x8_sse4);1894
+1895
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_2x8].p2s, filterPixelToShort_2x8_sse4);1896
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_2x16].p2s, filterPixelToShort_2x16_sse4);1897
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_4x4].p2s, filterPixelToShort_4x4_sse4);1898
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_4x8].p2s, filterPixelToShort_4x8_sse4);1899
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_4x16].p2s, filterPixelToShort_4x16_sse4);1900
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_4x32].p2s, filterPixelToShort_4x32_sse4);1901
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_6x16].p2s, filterPixelToShort_6x16_sse4);1902
1903
#if X86_641904
p.pelFilterLumaStrong[0] = PFX(pelFilterLumaStrong_V_sse4);1905
1906
p.cu[BLOCK_32x32].psy_cost_pp = PFX(psyCost_pp_32x32_avx2);1907
p.cu[BLOCK_64x64].psy_cost_pp = PFX(psyCost_pp_64x64_avx2);1908
1909
- p.pu[LUMA_8x4].addAvg = PFX(addAvg_8x4_avx2);1910
- p.pu[LUMA_8x8].addAvg = PFX(addAvg_8x8_avx2);1911
- p.pu[LUMA_8x16].addAvg = PFX(addAvg_8x16_avx2);1912
- p.pu[LUMA_8x32].addAvg = PFX(addAvg_8x32_avx2);1913
-1914
- p.pu[LUMA_12x16].addAvg = PFX(addAvg_12x16_avx2);1915
-1916
- p.pu[LUMA_16x4].addAvg = PFX(addAvg_16x4_avx2);1917
- p.pu[LUMA_16x8].addAvg = PFX(addAvg_16x8_avx2);1918
- p.pu[LUMA_16x12].addAvg = PFX(addAvg_16x12_avx2);1919
- p.pu[LUMA_16x16].addAvg = PFX(addAvg_16x16_avx2);1920
- p.pu[LUMA_16x32].addAvg = PFX(addAvg_16x32_avx2);1921
- p.pu[LUMA_16x64].addAvg = PFX(addAvg_16x64_avx2);1922
-1923
- p.pu[LUMA_24x32].addAvg = PFX(addAvg_24x32_avx2);1924
-1925
- p.pu[LUMA_32x8].addAvg = PFX(addAvg_32x8_avx2);1926
- p.pu[LUMA_32x16].addAvg = PFX(addAvg_32x16_avx2);1927
- p.pu[LUMA_32x24].addAvg = PFX(addAvg_32x24_avx2);1928
- p.pu[LUMA_32x32].addAvg = PFX(addAvg_32x32_avx2);1929
- p.pu[LUMA_32x64].addAvg = PFX(addAvg_32x64_avx2);1930
-1931
- p.pu[LUMA_48x64].addAvg = PFX(addAvg_48x64_avx2);1932
-1933
- p.pu[LUMA_64x16].addAvg = PFX(addAvg_64x16_avx2);1934
- p.pu[LUMA_64x32].addAvg = PFX(addAvg_64x32_avx2);1935
- p.pu[LUMA_64x48].addAvg = PFX(addAvg_64x48_avx2);1936
- p.pu[LUMA_64x64].addAvg = PFX(addAvg_64x64_avx2);1937
-1938
- p.chroma[X265_CSP_I420].pu[CHROMA_420_8x2].addAvg = PFX(addAvg_8x2_avx2);1939
- p.chroma[X265_CSP_I420].pu[CHROMA_420_8x4].addAvg = PFX(addAvg_8x4_avx2);1940
- p.chroma[X265_CSP_I420].pu[CHROMA_420_8x6].addAvg = PFX(addAvg_8x6_avx2);1941
- p.chroma[X265_CSP_I420].pu[CHROMA_420_8x8].addAvg = PFX(addAvg_8x8_avx2);1942
- p.chroma[X265_CSP_I420].pu[CHROMA_420_8x16].addAvg = PFX(addAvg_8x16_avx2);1943
- p.chroma[X265_CSP_I420].pu[CHROMA_420_8x32].addAvg = PFX(addAvg_8x32_avx2);1944
- p.chroma[X265_CSP_I420].pu[CHROMA_420_12x16].addAvg = PFX(addAvg_12x16_avx2);1945
- p.chroma[X265_CSP_I420].pu[CHROMA_420_16x4].addAvg = PFX(addAvg_16x4_avx2);1946
- p.chroma[X265_CSP_I420].pu[CHROMA_420_16x8].addAvg = PFX(addAvg_16x8_avx2);1947
- p.chroma[X265_CSP_I420].pu[CHROMA_420_16x12].addAvg = PFX(addAvg_16x12_avx2);1948
- p.chroma[X265_CSP_I420].pu[CHROMA_420_16x16].addAvg = PFX(addAvg_16x16_avx2);1949
- p.chroma[X265_CSP_I420].pu[CHROMA_420_16x32].addAvg = PFX(addAvg_16x32_avx2);1950
- p.chroma[X265_CSP_I420].pu[CHROMA_420_32x8].addAvg = PFX(addAvg_32x8_avx2);1951
- p.chroma[X265_CSP_I420].pu[CHROMA_420_32x16].addAvg = PFX(addAvg_32x16_avx2);1952
- p.chroma[X265_CSP_I420].pu[CHROMA_420_32x24].addAvg = PFX(addAvg_32x24_avx2);1953
- p.chroma[X265_CSP_I420].pu[CHROMA_420_32x32].addAvg = PFX(addAvg_32x32_avx2);1954
-1955
- p.chroma[X265_CSP_I422].pu[CHROMA_422_8x4].addAvg = PFX(addAvg_8x4_avx2);1956
- p.chroma[X265_CSP_I422].pu[CHROMA_422_8x8].addAvg = PFX(addAvg_8x8_avx2);1957
- p.chroma[X265_CSP_I422].pu[CHROMA_422_8x12].addAvg = PFX(addAvg_8x12_avx2);1958
- p.chroma[X265_CSP_I422].pu[CHROMA_422_8x16].addAvg = PFX(addAvg_8x16_avx2);1959
- p.chroma[X265_CSP_I422].pu[CHROMA_422_8x32].addAvg = PFX(addAvg_8x32_avx2);1960
- p.chroma[X265_CSP_I422].pu[CHROMA_422_8x64].addAvg = PFX(addAvg_8x64_avx2);1961
- p.chroma[X265_CSP_I422].pu[CHROMA_422_12x32].addAvg = PFX(addAvg_12x32_avx2);1962
- p.chroma[X265_CSP_I422].pu[CHROMA_422_16x8].addAvg = PFX(addAvg_16x8_avx2);1963
- p.chroma[X265_CSP_I422].pu[CHROMA_422_16x16].addAvg = PFX(addAvg_16x16_avx2);1964
- p.chroma[X265_CSP_I422].pu[CHROMA_422_16x24].addAvg = PFX(addAvg_16x24_avx2);1965
- p.chroma[X265_CSP_I422].pu[CHROMA_422_16x32].addAvg = PFX(addAvg_16x32_avx2);1966
- p.chroma[X265_CSP_I422].pu[CHROMA_422_16x64].addAvg = PFX(addAvg_16x64_avx2);1967
- p.chroma[X265_CSP_I422].pu[CHROMA_422_24x64].addAvg = PFX(addAvg_24x64_avx2);1968
- p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].addAvg = PFX(addAvg_32x16_avx2);1969
- p.chroma[X265_CSP_I422].pu[CHROMA_422_32x32].addAvg = PFX(addAvg_32x32_avx2);1970
- p.chroma[X265_CSP_I422].pu[CHROMA_422_32x48].addAvg = PFX(addAvg_32x48_avx2);1971
- p.chroma[X265_CSP_I422].pu[CHROMA_422_32x64].addAvg = PFX(addAvg_32x64_avx2);1972
+ ASSIGN2(p.pu[LUMA_8x4].addAvg, addAvg_8x4_avx2);1973
+ ASSIGN2(p.pu[LUMA_8x8].addAvg, addAvg_8x8_avx2);1974
+ ASSIGN2(p.pu[LUMA_8x16].addAvg, addAvg_8x16_avx2);1975
+ ASSIGN2(p.pu[LUMA_8x32].addAvg, addAvg_8x32_avx2);1976
+ ASSIGN2(p.pu[LUMA_12x16].addAvg, addAvg_12x16_avx2);1977
+ ASSIGN2(p.pu[LUMA_16x4].addAvg, addAvg_16x4_avx2);1978
+ ASSIGN2(p.pu[LUMA_16x8].addAvg, addAvg_16x8_avx2);1979
+ ASSIGN2(p.pu[LUMA_16x12].addAvg, addAvg_16x12_avx2);1980
+ ASSIGN2(p.pu[LUMA_16x16].addAvg, addAvg_16x16_avx2);1981
+ ASSIGN2(p.pu[LUMA_16x32].addAvg, addAvg_16x32_avx2);1982
+ ASSIGN2(p.pu[LUMA_16x64].addAvg, addAvg_16x64_avx2);1983
+ ASSIGN2(p.pu[LUMA_24x32].addAvg, addAvg_24x32_avx2);1984
+ ASSIGN2(p.pu[LUMA_32x8].addAvg, addAvg_32x8_avx2);1985
+ ASSIGN2(p.pu[LUMA_32x16].addAvg, addAvg_32x16_avx2);1986
+ ASSIGN2(p.pu[LUMA_32x24].addAvg, addAvg_32x24_avx2);1987
+ ASSIGN2(p.pu[LUMA_32x32].addAvg, addAvg_32x32_avx2);1988
+ ASSIGN2(p.pu[LUMA_32x64].addAvg, addAvg_32x64_avx2);1989
+ ASSIGN2(p.pu[LUMA_48x64].addAvg, addAvg_48x64_avx2);1990
+ ASSIGN2(p.pu[LUMA_64x16].addAvg, addAvg_64x16_avx2);1991
+ ASSIGN2(p.pu[LUMA_64x32].addAvg, addAvg_64x32_avx2);1992
+ ASSIGN2(p.pu[LUMA_64x48].addAvg, addAvg_64x48_avx2);1993
+ ASSIGN2(p.pu[LUMA_64x64].addAvg, addAvg_64x64_avx2);1994
+1995
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_8x2].addAvg, addAvg_8x2_avx2);1996
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_8x4].addAvg, addAvg_8x4_avx2);1997
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_8x6].addAvg, addAvg_8x6_avx2);1998
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_8x8].addAvg, addAvg_8x8_avx2);1999
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_8x16].addAvg, addAvg_8x16_avx2);2000
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_8x32].addAvg, addAvg_8x32_avx2);2001
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_12x16].addAvg, addAvg_12x16_avx2);2002
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_16x4].addAvg, addAvg_16x4_avx2);2003
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_16x8].addAvg, addAvg_16x8_avx2);2004
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_16x12].addAvg, addAvg_16x12_avx2);2005
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_16x16].addAvg, addAvg_16x16_avx2);2006
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_16x32].addAvg, addAvg_16x32_avx2);2007
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_32x8].addAvg, addAvg_32x8_avx2);2008
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_32x16].addAvg, addAvg_32x16_avx2);2009
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_32x24].addAvg, addAvg_32x24_avx2);2010
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_32x32].addAvg, addAvg_32x32_avx2);2011
+2012
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_8x4].addAvg, addAvg_8x4_avx2);2013
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_8x8].addAvg, addAvg_8x8_avx2);2014
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_8x12].addAvg, addAvg_8x12_avx2);2015
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_8x16].addAvg, addAvg_8x16_avx2);2016
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_8x32].addAvg, addAvg_8x32_avx2);2017
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_8x64].addAvg, addAvg_8x64_avx2);2018
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_12x32].addAvg, addAvg_12x32_avx2);2019
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_16x8].addAvg, addAvg_16x8_avx2);2020
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_16x16].addAvg, addAvg_16x16_avx2);2021
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_16x24].addAvg, addAvg_16x24_avx2);2022
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_16x32].addAvg, addAvg_16x32_avx2);2023
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_16x64].addAvg, addAvg_16x64_avx2);2024
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_24x64].addAvg, addAvg_24x64_avx2);2025
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].addAvg, addAvg_32x16_avx2);2026
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_32x32].addAvg, addAvg_32x32_avx2);2027
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_32x48].addAvg, addAvg_32x48_avx2);2028
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_32x64].addAvg, addAvg_32x64_avx2);2029
2030
p.cu[BLOCK_8x8].sa8d = PFX(pixel_sa8d_8x8_avx2);2031
p.cu[BLOCK_16x16].sa8d = PFX(pixel_sa8d_16x16_avx2);2032
2033
p.chroma[X265_CSP_I420].cu[BLOCK_420_16x16].sa8d = PFX(pixel_sa8d_16x16_avx2);2034
p.chroma[X265_CSP_I420].cu[BLOCK_420_32x32].sa8d = PFX(pixel_sa8d_32x32_avx2);2035
2036
- p.cu[BLOCK_16x16].add_ps = PFX(pixel_add_ps_16x16_avx2);2037
- p.cu[BLOCK_32x32].add_ps = PFX(pixel_add_ps_32x32_avx2);2038
- p.cu[BLOCK_64x64].add_ps = PFX(pixel_add_ps_64x64_avx2);2039
- p.chroma[X265_CSP_I420].cu[BLOCK_420_16x16].add_ps = PFX(pixel_add_ps_16x16_avx2);2040
- p.chroma[X265_CSP_I420].cu[BLOCK_420_32x32].add_ps = PFX(pixel_add_ps_32x32_avx2);2041
- p.chroma[X265_CSP_I422].cu[BLOCK_422_16x32].add_ps = PFX(pixel_add_ps_16x32_avx2);2042
- p.chroma[X265_CSP_I422].cu[BLOCK_422_32x64].add_ps = PFX(pixel_add_ps_32x64_avx2);2043
+ ASSIGN2(p.cu[BLOCK_16x16].add_ps, pixel_add_ps_16x16_avx2);2044
+ ASSIGN2(p.cu[BLOCK_32x32].add_ps, pixel_add_ps_32x32_avx2);2045
+ ASSIGN2(p.cu[BLOCK_64x64].add_ps, pixel_add_ps_64x64_avx2);2046
+ ASSIGN2(p.chroma[X265_CSP_I420].cu[BLOCK_420_16x16].add_ps, pixel_add_ps_16x16_avx2);2047
+ ASSIGN2(p.chroma[X265_CSP_I420].cu[BLOCK_420_32x32].add_ps, pixel_add_ps_32x32_avx2);2048
+ ASSIGN2(p.chroma[X265_CSP_I422].cu[BLOCK_422_16x32].add_ps, pixel_add_ps_16x32_avx2);2049
+ ASSIGN2(p.chroma[X265_CSP_I422].cu[BLOCK_422_32x64].add_ps, pixel_add_ps_32x64_avx2);2050
2051
p.cu[BLOCK_16x16].sub_ps = PFX(pixel_sub_ps_16x16_avx2);2052
p.cu[BLOCK_32x32].sub_ps = PFX(pixel_sub_ps_32x32_avx2);2053
2054
p.chroma[X265_CSP_I420].cu[BLOCK_420_32x32].sub_ps = PFX(pixel_sub_ps_32x32_avx2);2055
p.chroma[X265_CSP_I422].cu[BLOCK_422_16x32].sub_ps = PFX(pixel_sub_ps_16x32_avx2);2056
p.chroma[X265_CSP_I422].cu[BLOCK_422_32x64].sub_ps = PFX(pixel_sub_ps_32x64_avx2);2057
-2058
- p.pu[LUMA_16x4].pixelavg_pp = PFX(pixel_avg_16x4_avx2);2059
- p.pu[LUMA_16x8].pixelavg_pp = PFX(pixel_avg_16x8_avx2);2060
- p.pu[LUMA_16x12].pixelavg_pp = PFX(pixel_avg_16x12_avx2);2061
- p.pu[LUMA_16x16].pixelavg_pp = PFX(pixel_avg_16x16_avx2);2062
- p.pu[LUMA_16x32].pixelavg_pp = PFX(pixel_avg_16x32_avx2);2063
- p.pu[LUMA_16x64].pixelavg_pp = PFX(pixel_avg_16x64_avx2);2064
-2065
- p.pu[LUMA_32x64].pixelavg_pp = PFX(pixel_avg_32x64_avx2);2066
- p.pu[LUMA_32x32].pixelavg_pp = PFX(pixel_avg_32x32_avx2);2067
- p.pu[LUMA_32x24].pixelavg_pp = PFX(pixel_avg_32x24_avx2);2068
- p.pu[LUMA_32x16].pixelavg_pp = PFX(pixel_avg_32x16_avx2);2069
- p.pu[LUMA_32x8].pixelavg_pp = PFX(pixel_avg_32x8_avx2);2070
- p.pu[LUMA_48x64].pixelavg_pp = PFX(pixel_avg_48x64_avx2);2071
- p.pu[LUMA_64x64].pixelavg_pp = PFX(pixel_avg_64x64_avx2);2072
- p.pu[LUMA_64x48].pixelavg_pp = PFX(pixel_avg_64x48_avx2);2073
- p.pu[LUMA_64x32].pixelavg_pp = PFX(pixel_avg_64x32_avx2);2074
- p.pu[LUMA_64x16].pixelavg_pp = PFX(pixel_avg_64x16_avx2);2075
-2076
+ ASSIGN2(p.pu[LUMA_16x4].pixelavg_pp, pixel_avg_16x4_avx2);2077
+ ASSIGN2(p.pu[LUMA_16x8].pixelavg_pp, pixel_avg_16x8_avx2);2078
+ ASSIGN2(p.pu[LUMA_16x12].pixelavg_pp, pixel_avg_16x12_avx2);2079
+ ASSIGN2(p.pu[LUMA_16x16].pixelavg_pp, pixel_avg_16x16_avx2);2080
+ ASSIGN2(p.pu[LUMA_16x32].pixelavg_pp, pixel_avg_16x32_avx2);2081
+ ASSIGN2(p.pu[LUMA_16x64].pixelavg_pp, pixel_avg_16x64_avx2);2082
+2083
+ ASSIGN2(p.pu[LUMA_32x64].pixelavg_pp, pixel_avg_32x64_avx2);2084
+ ASSIGN2(p.pu[LUMA_32x32].pixelavg_pp, pixel_avg_32x32_avx2);2085
+ ASSIGN2(p.pu[LUMA_32x24].pixelavg_pp, pixel_avg_32x24_avx2);2086
+ ASSIGN2(p.pu[LUMA_32x16].pixelavg_pp, pixel_avg_32x16_avx2);2087
+ ASSIGN2(p.pu[LUMA_32x8].pixelavg_pp, pixel_avg_32x8_avx2);2088
+ ASSIGN2(p.pu[LUMA_48x64].pixelavg_pp, pixel_avg_48x64_avx2);2089
+ ASSIGN2(p.pu[LUMA_64x64].pixelavg_pp, pixel_avg_64x64_avx2);2090
+ ASSIGN2(p.pu[LUMA_64x48].pixelavg_pp, pixel_avg_64x48_avx2);2091
+ ASSIGN2(p.pu[LUMA_64x32].pixelavg_pp, pixel_avg_64x32_avx2);2092
+ ASSIGN2(p.pu[LUMA_64x16].pixelavg_pp, pixel_avg_64x16_avx2);2093
p.pu[LUMA_16x16].satd = PFX(pixel_satd_16x16_avx2);2094
p.pu[LUMA_16x8].satd = PFX(pixel_satd_16x8_avx2);2095
p.pu[LUMA_8x16].satd = PFX(pixel_satd_8x16_avx2);2096
2097
p.chroma[X265_CSP_I420].cu[BLOCK_420_16x16].sse_pp = PFX(pixel_ssd_16x16_avx2);2098
p.chroma[X265_CSP_I420].cu[BLOCK_420_32x32].sse_pp = PFX(pixel_ssd_32x32_avx2);2099
2100
- p.cu[BLOCK_16x16].ssd_s = PFX(pixel_ssd_s_16_avx2);2101
- p.cu[BLOCK_32x32].ssd_s = PFX(pixel_ssd_s_32_avx2);2102
-2103
+ ASSIGN2(p.cu[BLOCK_16x16].ssd_s, pixel_ssd_s_16_avx2);2104
+ ASSIGN2(p.cu[BLOCK_32x32].ssd_s, pixel_ssd_s_32_avx2);2105
p.cu[BLOCK_8x8].copy_cnt = PFX(copy_cnt_8_avx2);2106
p.cu[BLOCK_16x16].copy_cnt = PFX(copy_cnt_16_avx2);2107
p.cu[BLOCK_32x32].copy_cnt = PFX(copy_cnt_32_avx2);2108
-2109
- p.cu[BLOCK_16x16].blockfill_s = PFX(blockfill_s_16x16_avx2);2110
- p.cu[BLOCK_32x32].blockfill_s = PFX(blockfill_s_32x32_avx2);2111
-2112
- ALL_LUMA_TU_S(cpy1Dto2D_shl, cpy1Dto2D_shl_, avx2);2113
+ ASSIGN2(p.cu[BLOCK_16x16].blockfill_s, blockfill_s_16x16_avx2);2114
+ ALL_LUMA_TU_S(cpy1Dto2D_shl[ALIGNED], cpy1Dto2D_shl_, avx2);2115
+ ALL_LUMA_TU_S(cpy1Dto2D_shl[NONALIGNED], cpy1Dto2D_shl_, avx2);2116
ALL_LUMA_TU_S(cpy1Dto2D_shr, cpy1Dto2D_shr_, avx2);2117
-2118
p.cu[BLOCK_8x8].cpy2Dto1D_shl = PFX(cpy2Dto1D_shl_8_avx2);2119
p.cu[BLOCK_16x16].cpy2Dto1D_shl = PFX(cpy2Dto1D_shl_16_avx2);2120
p.cu[BLOCK_32x32].cpy2Dto1D_shl = PFX(cpy2Dto1D_shl_32_avx2);2121
2122
p.dequant_normal = PFX(dequant_normal_avx2);2123
p.dequant_scaling = PFX(dequant_scaling_avx2);2124
2125
- p.cu[BLOCK_16x16].calcresidual = PFX(getResidual16_avx2);2126
- p.cu[BLOCK_32x32].calcresidual = PFX(getResidual32_avx2);2127
+ ASSIGN2(p.cu[BLOCK_16x16].calcresidual, getResidual16_avx2);2128
+ ASSIGN2(p.cu[BLOCK_32x32].calcresidual, getResidual32_avx2);2129
2130
- p.scale1D_128to64 = PFX(scale1D_128to64_avx2);2131
+ ASSIGN2(p.scale1D_128to64, scale1D_128to64_avx2);2132
p.weight_pp = PFX(weight_pp_avx2);2133
p.weight_sp = PFX(weight_sp_avx2);2134
2135
2136
ALL_LUMA_PU_T(luma_hvpp, interp_8tap_hv_pp_cpu);2137
p.pu[LUMA_4x4].luma_hvpp = interp_8tap_hv_pp_cpu<LUMA_4x4>;2138
2139
- p.pu[LUMA_16x4].convert_p2s = PFX(filterPixelToShort_16x4_avx2);2140
- p.pu[LUMA_16x8].convert_p2s = PFX(filterPixelToShort_16x8_avx2);2141
- p.pu[LUMA_16x12].convert_p2s = PFX(filterPixelToShort_16x12_avx2);2142
- p.pu[LUMA_16x16].convert_p2s = PFX(filterPixelToShort_16x16_avx2);2143
- p.pu[LUMA_16x32].convert_p2s = PFX(filterPixelToShort_16x32_avx2);2144
- p.pu[LUMA_16x64].convert_p2s = PFX(filterPixelToShort_16x64_avx2);2145
- p.pu[LUMA_32x8].convert_p2s = PFX(filterPixelToShort_32x8_avx2);2146
- p.pu[LUMA_32x16].convert_p2s = PFX(filterPixelToShort_32x16_avx2);2147
- p.pu[LUMA_32x24].convert_p2s = PFX(filterPixelToShort_32x24_avx2);2148
- p.pu[LUMA_32x32].convert_p2s = PFX(filterPixelToShort_32x32_avx2);2149
- p.pu[LUMA_32x64].convert_p2s = PFX(filterPixelToShort_32x64_avx2);2150
- p.pu[LUMA_64x16].convert_p2s = PFX(filterPixelToShort_64x16_avx2);2151
- p.pu[LUMA_64x32].convert_p2s = PFX(filterPixelToShort_64x32_avx2);2152
- p.pu[LUMA_64x48].convert_p2s = PFX(filterPixelToShort_64x48_avx2);2153
- p.pu[LUMA_64x64].convert_p2s = PFX(filterPixelToShort_64x64_avx2);2154
- p.pu[LUMA_48x64].convert_p2s = PFX(filterPixelToShort_48x64_avx2);2155
- p.pu[LUMA_24x32].convert_p2s = PFX(filterPixelToShort_24x32_avx2);2156
-2157
- p.chroma[X265_CSP_I420].pu[CHROMA_420_16x4].p2s = PFX(filterPixelToShort_16x4_avx2);2158
- p.chroma[X265_CSP_I420].pu[CHROMA_420_16x8].p2s = PFX(filterPixelToShort_16x8_avx2);2159
- p.chroma[X265_CSP_I420].pu[CHROMA_420_16x12].p2s = PFX(filterPixelToShort_16x12_avx2);2160
- p.chroma[X265_CSP_I420].pu[CHROMA_420_16x16].p2s = PFX(filterPixelToShort_16x16_avx2);2161
- p.chroma[X265_CSP_I420].pu[CHROMA_420_16x32].p2s = PFX(filterPixelToShort_16x32_avx2);2162
- p.chroma[X265_CSP_I420].pu[CHROMA_420_24x32].p2s = PFX(filterPixelToShort_24x32_avx2);2163
- p.chroma[X265_CSP_I420].pu[CHROMA_420_32x8].p2s = PFX(filterPixelToShort_32x8_avx2);2164
- p.chroma[X265_CSP_I420].pu[CHROMA_420_32x16].p2s = PFX(filterPixelToShort_32x16_avx2);2165
- p.chroma[X265_CSP_I420].pu[CHROMA_420_32x24].p2s = PFX(filterPixelToShort_32x24_avx2);2166
- p.chroma[X265_CSP_I420].pu[CHROMA_420_32x32].p2s = PFX(filterPixelToShort_32x32_avx2);2167
- p.chroma[X265_CSP_I422].pu[CHROMA_422_16x8].p2s = PFX(filterPixelToShort_16x8_avx2);2168
- p.chroma[X265_CSP_I422].pu[CHROMA_422_16x16].p2s = PFX(filterPixelToShort_16x16_avx2);2169
- p.chroma[X265_CSP_I422].pu[CHROMA_422_16x24].p2s = PFX(filterPixelToShort_16x24_avx2);2170
- p.chroma[X265_CSP_I422].pu[CHROMA_422_16x32].p2s = PFX(filterPixelToShort_16x32_avx2);2171
- p.chroma[X265_CSP_I422].pu[CHROMA_422_16x64].p2s = PFX(filterPixelToShort_16x64_avx2);2172
- p.chroma[X265_CSP_I422].pu[CHROMA_422_24x64].p2s = PFX(filterPixelToShort_24x64_avx2);2173
- p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].p2s = PFX(filterPixelToShort_32x16_avx2);2174
- p.chroma[X265_CSP_I422].pu[CHROMA_422_32x32].p2s = PFX(filterPixelToShort_32x32_avx2);2175
- p.chroma[X265_CSP_I422].pu[CHROMA_422_32x48].p2s = PFX(filterPixelToShort_32x48_avx2);2176
- p.chroma[X265_CSP_I422].pu[CHROMA_422_32x64].p2s = PFX(filterPixelToShort_32x64_avx2);2177
+ ASSIGN2(p.pu[LUMA_16x4].convert_p2s, filterPixelToShort_16x4_avx2);2178
+ ASSIGN2(p.pu[LUMA_16x8].convert_p2s, filterPixelToShort_16x8_avx2);2179
+ ASSIGN2(p.pu[LUMA_16x12].convert_p2s, filterPixelToShort_16x12_avx2);2180
+ ASSIGN2(p.pu[LUMA_16x16].convert_p2s, filterPixelToShort_16x16_avx2);2181
+ ASSIGN2(p.pu[LUMA_16x32].convert_p2s, filterPixelToShort_16x32_avx2);2182
+ ASSIGN2(p.pu[LUMA_16x64].convert_p2s, filterPixelToShort_16x64_avx2);2183
+ ASSIGN2(p.pu[LUMA_32x8].convert_p2s, filterPixelToShort_32x8_avx2);2184
+ ASSIGN2(p.pu[LUMA_32x16].convert_p2s, filterPixelToShort_32x16_avx2);2185
+ ASSIGN2(p.pu[LUMA_32x24].convert_p2s, filterPixelToShort_32x24_avx2);2186
+ ASSIGN2(p.pu[LUMA_32x32].convert_p2s, filterPixelToShort_32x32_avx2);2187
+ ASSIGN2(p.pu[LUMA_32x64].convert_p2s, filterPixelToShort_32x64_avx2);2188
+ ASSIGN2(p.pu[LUMA_64x16].convert_p2s, filterPixelToShort_64x16_avx2);2189
+ ASSIGN2(p.pu[LUMA_64x32].convert_p2s, filterPixelToShort_64x32_avx2);2190
+ ASSIGN2(p.pu[LUMA_64x48].convert_p2s, filterPixelToShort_64x48_avx2);2191
+ ASSIGN2(p.pu[LUMA_64x64].convert_p2s, filterPixelToShort_64x64_avx2);2192
+ ASSIGN2(p.pu[LUMA_48x64].convert_p2s, filterPixelToShort_48x64_avx2);2193
+ ASSIGN2(p.pu[LUMA_24x32].convert_p2s, filterPixelToShort_24x32_avx2);2194
+2195
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_16x4].p2s, filterPixelToShort_16x4_avx2);2196
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_16x8].p2s, filterPixelToShort_16x8_avx2);2197
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_16x12].p2s, filterPixelToShort_16x12_avx2);2198
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_16x16].p2s, filterPixelToShort_16x16_avx2);2199
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_16x32].p2s, filterPixelToShort_16x32_avx2);2200
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_24x32].p2s, filterPixelToShort_24x32_avx2);2201
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_32x8].p2s, filterPixelToShort_32x8_avx2);2202
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_32x16].p2s, filterPixelToShort_32x16_avx2);2203
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_32x24].p2s, filterPixelToShort_32x24_avx2);2204
+ ASSIGN2(p.chroma[X265_CSP_I420].pu[CHROMA_420_32x32].p2s, filterPixelToShort_32x32_avx2);2205
+2206
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_16x8].p2s, filterPixelToShort_16x8_avx2);2207
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_16x16].p2s, filterPixelToShort_16x16_avx2);2208
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_16x24].p2s, filterPixelToShort_16x24_avx2);2209
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_16x32].p2s, filterPixelToShort_16x32_avx2);2210
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_16x64].p2s, filterPixelToShort_16x64_avx2);2211
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_24x64].p2s, filterPixelToShort_24x64_avx2);2212
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].p2s, filterPixelToShort_32x16_avx2);2213
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_32x32].p2s, filterPixelToShort_32x32_avx2);2214
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_32x48].p2s, filterPixelToShort_32x48_avx2);2215
+ ASSIGN2(p.chroma[X265_CSP_I422].pu[CHROMA_422_32x64].p2s, filterPixelToShort_32x64_avx2);2216
2217
//i422 for chroma_hpp2218
p.chroma[X265_CSP_I422].pu[CHROMA_422_12x32].filter_hpp = PFX(interp_4tap_horiz_pp_12x32_avx2);2219
2220
p.integral_inith[INTEGRAL_16] = PFX(integral16h_avx2);2221
p.integral_inith[INTEGRAL_24] = PFX(integral24h_avx2);2222
p.integral_inith[INTEGRAL_32] = PFX(integral32h_avx2);2223
+ p.cu[BLOCK_4x4].nonPsyRdoQuant = PFX(nonPsyRdoQuant4_avx2);2224
+ p.cu[BLOCK_8x8].nonPsyRdoQuant = PFX(nonPsyRdoQuant8_avx2);2225
+ p.cu[BLOCK_16x16].nonPsyRdoQuant = PFX(nonPsyRdoQuant16_avx2);2226
+ p.cu[BLOCK_32x32].nonPsyRdoQuant = PFX(nonPsyRdoQuant32_avx2);2227
+ p.cu[BLOCK_4x4].psyRdoQuant_1p = PFX(psyRdoQuant_1p4_avx2);2228
+ p.cu[BLOCK_8x8].psyRdoQuant_1p = PFX(psyRdoQuant_1p8_avx2);2229
+ p.cu[BLOCK_16x16].psyRdoQuant_1p = PFX(psyRdoQuant_1p16_avx2);2230
+ p.cu[BLOCK_32x32].psyRdoQuant_1p = PFX(psyRdoQuant_1p32_avx2);2231
+2232
+ }2233
+ if (cpuMask & X265_CPU_AVX512)2234
+ {2235
+ p.pu[LUMA_32x8].sad = PFX(pixel_sad_32x8_avx512);2236
+ // p.pu[LUMA_32x16].sad = PFX(pixel_sad_32x16_avx512);2237
+ p.pu[LUMA_32x24].sad = PFX(pixel_sad_32x24_avx512);2238
+ p.pu[LUMA_32x32].sad = PFX(pixel_sad_32x32_avx512);2239
+ //p.pu[LUMA_32x64].sad = PFX(pixel_sad_32x64_avx512);2240
+ p.pu[LUMA_64x16].sad = PFX(pixel_sad_64x16_avx512);2241
+ p.pu[LUMA_64x32].sad = PFX(pixel_sad_64x32_avx512);2242
+ p.pu[LUMA_64x48].sad = PFX(pixel_sad_64x48_avx512);2243
+ p.pu[LUMA_64x64].sad = PFX(pixel_sad_64x64_avx512);2244
+2245
+ p.pu[LUMA_32x8].sad_x3 = PFX(pixel_sad_x3_32x8_avx512);2246
+ p.pu[LUMA_32x16].sad_x3 = PFX(pixel_sad_x3_32x16_avx512);2247
+ p.pu[LUMA_32x24].sad_x3 = PFX(pixel_sad_x3_32x24_avx512);2248
+ p.pu[LUMA_32x32].sad_x3 = PFX(pixel_sad_x3_32x32_avx512);2249
+ p.pu[LUMA_32x64].sad_x3 = PFX(pixel_sad_x3_32x64_avx512);2250
+ p.pu[LUMA_64x16].sad_x3 = PFX(pixel_sad_x3_64x16_avx512);2251
+ p.pu[LUMA_64x32].sad_x3 = PFX(pixel_sad_x3_64x32_avx512);2252
+ p.pu[LUMA_64x48].sad_x3 = PFX(pixel_sad_x3_64x48_avx512);2253
+ p.pu[LUMA_64x64].sad_x3 = PFX(pixel_sad_x3_64x64_avx512);2254
+ p.pu[LUMA_48x64].sad_x3 = PFX(pixel_sad_x3_48x64_avx512);2255
+2256
+ p.pu[LUMA_32x32].sad_x4 = PFX(pixel_sad_x4_32x32_avx512);2257
+ p.pu[LUMA_32x16].sad_x4 = PFX(pixel_sad_x4_32x16_avx512);2258
+ p.pu[LUMA_32x64].sad_x4 = PFX(pixel_sad_x4_32x64_avx512);2259
+ p.pu[LUMA_32x24].sad_x4 = PFX(pixel_sad_x4_32x24_avx512);2260
+ p.pu[LUMA_32x8].sad_x4 = PFX(pixel_sad_x4_32x8_avx512);2261
+ p.pu[LUMA_64x16].sad_x4 = PFX(pixel_sad_x4_64x16_avx512);2262
+ p.pu[LUMA_64x32].sad_x4 = PFX(pixel_sad_x4_64x32_avx512);2263
+ p.pu[LUMA_64x48].sad_x4 = PFX(pixel_sad_x4_64x48_avx512);2264
+ p.pu[LUMA_64x64].sad_x4 = PFX(pixel_sad_x4_64x64_avx512);2265
+ p.pu[LUMA_48x64].sad_x4 = PFX(pixel_sad_x4_48x64_avx512);2266
+2267
+ p.pu[LUMA_4x4].satd = PFX(pixel_satd_4x4_avx512);2268
+ p.pu[LUMA_4x8].satd = PFX(pixel_satd_4x8_avx512);2269
+ p.pu[LUMA_4x16].satd = PFX(pixel_satd_4x16_avx512);2270
+ p.pu[LUMA_8x4].satd = PFX(pixel_satd_8x4_avx512);2271
+ p.pu[LUMA_8x8].satd = PFX(pixel_satd_8x8_avx512);2272
+ p.pu[LUMA_8x16].satd = PFX(pixel_satd_8x16_avx512);2273
+ p.pu[LUMA_16x8].satd = PFX(pixel_satd_16x8_avx512);2274
+ p.pu[LUMA_16x16].satd = PFX(pixel_satd_16x16_avx512);2275
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_4x4].satd = PFX(pixel_satd_4x4_avx512);2276
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_4x8].satd = PFX(pixel_satd_4x8_avx512);2277
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_4x16].satd = PFX(pixel_satd_4x16_avx512);2278
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x4].satd = PFX(pixel_satd_8x4_avx512);2279
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x8].satd = PFX(pixel_satd_8x8_avx512);2280
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x16].satd = PFX(pixel_satd_8x16_avx512);2281
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x8].satd = PFX(pixel_satd_16x8_avx512);2282
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x16].satd = PFX(pixel_satd_16x16_avx512);2283
+2284
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_4x4].satd = PFX(pixel_satd_4x4_avx512);2285
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_4x8].satd = PFX(pixel_satd_4x8_avx512);2286
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_4x16].satd = PFX(pixel_satd_4x16_avx512);2287
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_8x4].satd = PFX(pixel_satd_8x4_avx512);2288
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_8x8].satd = PFX(pixel_satd_8x8_avx512);2289
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_8x16].satd = PFX(pixel_satd_8x16_avx512);2290
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x8].satd = PFX(pixel_satd_16x8_avx512);2291
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x16].satd = PFX(pixel_satd_16x16_avx512);2292
+2293
+ p.cu[BLOCK_8x8].sa8d = PFX(pixel_sa8d_8x8_avx512);2294
+ p.chroma[X265_CSP_I420].cu[BLOCK_420_8x8].sa8d = PFX(pixel_sa8d_8x8_avx512);2295
+2296
+ p.cu[BLOCK_8x8].var = PFX(pixel_var_8x8_avx512);2297
+ p.cu[BLOCK_16x16].var = PFX(pixel_var_16x16_avx512);2298
+ p.cu[BLOCK_32x32].var = PFX(pixel_var_32x32_avx512);2299
+ p.cu[BLOCK_64x64].var = PFX(pixel_var_64x64_avx512);2300
+ ASSIGN2(p.pu[LUMA_16x64].pixelavg_pp, pixel_avg_16x64_avx512);2301
+ ASSIGN2(p.pu[LUMA_16x32].pixelavg_pp, pixel_avg_16x32_avx512);2302
+ ASSIGN2(p.pu[LUMA_16x16].pixelavg_pp, pixel_avg_16x16_avx512);2303
+ ASSIGN2(p.pu[LUMA_16x12].pixelavg_pp, pixel_avg_16x12_avx512);2304
+ ASSIGN2(p.pu[LUMA_16x8].pixelavg_pp, pixel_avg_16x8_avx512);2305
+ ASSIGN2(p.pu[LUMA_16x4].pixelavg_pp, pixel_avg_16x4_avx512);2306
+ ASSIGN2(p.pu[LUMA_8x32].pixelavg_pp, pixel_avg_8x32_avx512);2307
+ ASSIGN2(p.pu[LUMA_8x16].pixelavg_pp, pixel_avg_8x16_avx512);2308
+ ASSIGN2(p.pu[LUMA_8x8].pixelavg_pp, pixel_avg_8x8_avx512);2309
+ //p.pu[LUMA_8x4].pixelavg_pp = PFX(pixel_avg_8x4_avx512);2310
+ p.pu[LUMA_4x4].sad = PFX(pixel_sad_4x4_avx512);2311
+ p.pu[LUMA_4x8].sad = PFX(pixel_sad_4x8_avx512);2312
+ p.pu[LUMA_4x16].sad = PFX(pixel_sad_4x16_avx512);2313
+ p.pu[LUMA_8x4].sad = PFX(pixel_sad_8x4_avx512);2314
+ p.pu[LUMA_8x8].sad = PFX(pixel_sad_8x8_avx512);2315
+ // p.pu[LUMA_8x16].sad = PFX(pixel_sad_8x16_avx512);2316
+ p.pu[LUMA_16x8].sad = PFX(pixel_sad_16x8_avx512);2317
+ p.pu[LUMA_16x16].sad = PFX(pixel_sad_16x16_avx512);2318
+2319
+ p.pu[LUMA_64x64].copy_pp = PFX(blockcopy_pp_64x64_avx512);2320
+ p.pu[LUMA_64x32].copy_pp = PFX(blockcopy_pp_64x32_avx512);2321
+ p.pu[LUMA_64x48].copy_pp = PFX(blockcopy_pp_64x48_avx512);2322
+ p.pu[LUMA_64x16].copy_pp = PFX(blockcopy_pp_64x16_avx512);2323
+ p.pu[LUMA_32x16].copy_pp = PFX(blockcopy_pp_32x16_avx512);2324
+ p.pu[LUMA_32x24].copy_pp = PFX(blockcopy_pp_32x24_avx512);2325
+ p.pu[LUMA_32x32].copy_pp = PFX(blockcopy_pp_32x32_avx512);2326
+ p.pu[LUMA_32x64].copy_pp = PFX(blockcopy_pp_32x64_avx512);2327
+2328
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x16].copy_pp = PFX(blockcopy_pp_32x16_avx512);2329
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x24].copy_pp = PFX(blockcopy_pp_32x24_avx512);2330
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x32].copy_pp = PFX(blockcopy_pp_32x32_avx512);2331
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].copy_pp = PFX(blockcopy_pp_32x16_avx512);2332
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x32].copy_pp = PFX(blockcopy_pp_32x32_avx512);2333
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x48].copy_pp = PFX(blockcopy_pp_32x48_avx512);2334
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x64].copy_pp = PFX(blockcopy_pp_32x64_avx512);2335
+2336
+ p.cu[BLOCK_64x64].copy_sp = PFX(blockcopy_sp_64x64_avx512);2337
+ p.cu[BLOCK_32x32].copy_sp = PFX(blockcopy_sp_32x32_avx512);2338
+ p.chroma[X265_CSP_I420].cu[BLOCK_420_32x32].copy_sp = PFX(blockcopy_sp_32x32_avx512);2339
+ p.chroma[X265_CSP_I422].cu[BLOCK_422_32x64].copy_sp = PFX(blockcopy_sp_32x64_avx512);2340
+2341
+ p.cu[BLOCK_32x32].copy_ps = PFX(blockcopy_ps_32x32_avx512);2342
+ p.chroma[X265_CSP_I420].cu[CHROMA_420_32x32].copy_ps = PFX(blockcopy_ps_32x32_avx512);2343
+ p.chroma[X265_CSP_I422].cu[CHROMA_422_32x64].copy_ps = PFX(blockcopy_ps_32x64_avx512);2344
+ p.cu[BLOCK_64x64].copy_ps = PFX(blockcopy_ps_64x64_avx512);2345
+2346
+ p.scale1D_128to64[NONALIGNED] = PFX(scale1D_128to64_avx512);2347
+ p.scale1D_128to64[ALIGNED] = PFX(scale1D_128to64_aligned_avx512);2348
+2349
+ p.pu[LUMA_64x16].addAvg[NONALIGNED] = PFX(addAvg_64x16_avx512);2350
+ p.pu[LUMA_64x32].addAvg[NONALIGNED] = PFX(addAvg_64x32_avx512);2351
+ p.pu[LUMA_64x48].addAvg[NONALIGNED] = PFX(addAvg_64x48_avx512);2352
+ p.pu[LUMA_64x64].addAvg[NONALIGNED] = PFX(addAvg_64x64_avx512);2353
+ p.pu[LUMA_32x8].addAvg[NONALIGNED] = PFX(addAvg_32x8_avx512);2354
+ p.pu[LUMA_32x16].addAvg[NONALIGNED] = PFX(addAvg_32x16_avx512);2355
+ p.pu[LUMA_32x24].addAvg[NONALIGNED] = PFX(addAvg_32x24_avx512);2356
+ p.pu[LUMA_32x32].addAvg[NONALIGNED] = PFX(addAvg_32x32_avx512);2357
+ p.pu[LUMA_32x64].addAvg[NONALIGNED] = PFX(addAvg_32x64_avx512);2358
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x8].addAvg[NONALIGNED] = PFX(addAvg_32x8_avx512);2359
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x16].addAvg[NONALIGNED] = PFX(addAvg_32x16_avx512);2360
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x24].addAvg[NONALIGNED] = PFX(addAvg_32x24_avx512);2361
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x32].addAvg[NONALIGNED] = PFX(addAvg_32x32_avx512);2362
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].addAvg[NONALIGNED] = PFX(addAvg_32x16_avx512);2363
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x48].addAvg[NONALIGNED] = PFX(addAvg_32x48_avx512);2364
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x64].addAvg[NONALIGNED] = PFX(addAvg_32x64_avx512);2365
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x32].addAvg[NONALIGNED] = PFX(addAvg_32x32_avx512);2366
+2367
+ p.pu[LUMA_32x8].addAvg[ALIGNED] = PFX(addAvg_aligned_32x8_avx512);2368
+ p.pu[LUMA_32x16].addAvg[ALIGNED] = PFX(addAvg_aligned_32x16_avx512);2369
+ p.pu[LUMA_32x24].addAvg[ALIGNED] = PFX(addAvg_aligned_32x24_avx512);2370
+ p.pu[LUMA_32x32].addAvg[ALIGNED] = PFX(addAvg_aligned_32x32_avx512);2371
+ p.pu[LUMA_32x64].addAvg[ALIGNED] = PFX(addAvg_aligned_32x64_avx512);2372
+ p.pu[LUMA_64x16].addAvg[ALIGNED] = PFX(addAvg_aligned_64x16_avx512);2373
+ p.pu[LUMA_64x32].addAvg[ALIGNED] = PFX(addAvg_aligned_64x32_avx512);2374
+ p.pu[LUMA_64x48].addAvg[ALIGNED] = PFX(addAvg_aligned_64x48_avx512);2375
+ p.pu[LUMA_64x64].addAvg[ALIGNED] = PFX(addAvg_aligned_64x64_avx512);2376
+2377
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x8].addAvg[ALIGNED] = PFX(addAvg_aligned_32x8_avx512);2378
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x16].addAvg[ALIGNED] = PFX(addAvg_aligned_32x16_avx512);2379
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x24].addAvg[ALIGNED] = PFX(addAvg_aligned_32x24_avx512);2380
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x32].addAvg[ALIGNED] = PFX(addAvg_aligned_32x32_avx512);2381
+2382
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].addAvg[ALIGNED] = PFX(addAvg_aligned_32x16_avx512);2383
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x48].addAvg[ALIGNED] = PFX(addAvg_aligned_32x48_avx512);2384
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x64].addAvg[ALIGNED] = PFX(addAvg_aligned_32x64_avx512);2385
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x32].addAvg[ALIGNED] = PFX(addAvg_aligned_32x32_avx512);2386
+2387
+ p.cu[BLOCK_32x32].blockfill_s[NONALIGNED] = PFX(blockfill_s_32x32_avx512);2388
+ p.cu[BLOCK_32x32].blockfill_s[ALIGNED] = PFX(blockfill_s_aligned_32x32_avx512);2389
+2390
+ p.cu[BLOCK_64x64].add_ps[NONALIGNED] = PFX(pixel_add_ps_64x64_avx512);2391
+ p.cu[BLOCK_32x32].add_ps[NONALIGNED] = PFX(pixel_add_ps_32x32_avx512);2392
+ p.chroma[X265_CSP_I420].cu[BLOCK_420_32x32].add_ps[NONALIGNED] = PFX(pixel_add_ps_32x32_avx512);2393
+ p.chroma[X265_CSP_I422].cu[BLOCK_422_32x64].add_ps[NONALIGNED] = PFX(pixel_add_ps_32x64_avx512);2394
+2395
+ p.cu[BLOCK_32x32].add_ps[ALIGNED] = PFX(pixel_add_ps_aligned_32x32_avx512);2396
+ p.cu[BLOCK_64x64].add_ps[ALIGNED] = PFX(pixel_add_ps_aligned_64x64_avx512);2397
+ p.chroma[X265_CSP_I420].cu[BLOCK_420_32x32].add_ps[ALIGNED] = PFX(pixel_add_ps_aligned_32x32_avx512);2398
+ p.chroma[X265_CSP_I422].cu[BLOCK_422_32x64].add_ps[ALIGNED] = PFX(pixel_add_ps_aligned_32x64_avx512);2399
+2400
+ p.cu[BLOCK_64x64].sub_ps = PFX(pixel_sub_ps_64x64_avx512);2401
+ p.cu[BLOCK_32x32].sub_ps = PFX(pixel_sub_ps_32x32_avx512);2402
+ p.chroma[X265_CSP_I420].cu[BLOCK_420_32x32].sub_ps = PFX(pixel_sub_ps_32x32_avx512);2403
+ p.chroma[X265_CSP_I422].cu[BLOCK_422_32x64].sub_ps = PFX(pixel_sub_ps_32x64_avx512);2404
+2405
+ p.pu[LUMA_64x16].convert_p2s[NONALIGNED] = PFX(filterPixelToShort_64x16_avx512);2406
+ p.pu[LUMA_64x32].convert_p2s[NONALIGNED] = PFX(filterPixelToShort_64x32_avx512);2407
+ p.pu[LUMA_64x48].convert_p2s[NONALIGNED] = PFX(filterPixelToShort_64x48_avx512);2408
+ p.pu[LUMA_64x64].convert_p2s[NONALIGNED] = PFX(filterPixelToShort_64x64_avx512);2409
+ p.pu[LUMA_32x8].convert_p2s[NONALIGNED] = PFX(filterPixelToShort_32x8_avx2);2410
+ p.pu[LUMA_32x16].convert_p2s[NONALIGNED] = PFX(filterPixelToShort_32x16_avx512);2411
+ p.pu[LUMA_32x24].convert_p2s[NONALIGNED] = PFX(filterPixelToShort_32x24_avx512);2412
+ p.pu[LUMA_32x32].convert_p2s[NONALIGNED] = PFX(filterPixelToShort_32x32_avx512);2413
+ p.pu[LUMA_32x64].convert_p2s[NONALIGNED] = PFX(filterPixelToShort_32x64_avx512);2414
+ p.pu[LUMA_48x64].convert_p2s[NONALIGNED] = PFX(filterPixelToShort_48x64_avx512);2415
+2416
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x8].p2s[NONALIGNED] = PFX(filterPixelToShort_32x8_avx512);2417
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x16].p2s[NONALIGNED] = PFX(filterPixelToShort_32x16_avx512);2418
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x24].p2s[NONALIGNED] = PFX(filterPixelToShort_32x24_avx512);2419
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x32].p2s[NONALIGNED] = PFX(filterPixelToShort_32x32_avx512);2420
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].p2s[NONALIGNED] = PFX(filterPixelToShort_32x16_avx512);2421
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x32].p2s[NONALIGNED] = PFX(filterPixelToShort_32x32_avx512);2422
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x48].p2s[NONALIGNED] = PFX(filterPixelToShort_32x48_avx512);2423
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x64].p2s[NONALIGNED] = PFX(filterPixelToShort_32x64_avx512);2424
+ p.chroma[X265_CSP_I444].pu[LUMA_32x8].p2s[NONALIGNED] = PFX(filterPixelToShort_32x8_avx2);2425
+ p.chroma[X265_CSP_I444].pu[LUMA_32x16].p2s[NONALIGNED] = PFX(filterPixelToShort_32x16_avx512);2426
+ p.chroma[X265_CSP_I444].pu[LUMA_32x24].p2s[NONALIGNED] = PFX(filterPixelToShort_32x24_avx512);2427
+ p.chroma[X265_CSP_I444].pu[LUMA_32x32].p2s[NONALIGNED] = PFX(filterPixelToShort_32x32_avx512);2428
+ p.chroma[X265_CSP_I444].pu[LUMA_32x64].p2s[NONALIGNED] = PFX(filterPixelToShort_32x64_avx512);2429
+ p.chroma[X265_CSP_I444].pu[LUMA_64x16].p2s[NONALIGNED] = PFX(filterPixelToShort_64x16_avx512);2430
+ p.chroma[X265_CSP_I444].pu[LUMA_64x32].p2s[NONALIGNED] = PFX(filterPixelToShort_64x32_avx512);2431
+ p.chroma[X265_CSP_I444].pu[LUMA_64x48].p2s[NONALIGNED] = PFX(filterPixelToShort_64x48_avx512);2432
+ p.chroma[X265_CSP_I444].pu[LUMA_64x64].p2s[NONALIGNED] = PFX(filterPixelToShort_64x64_avx512);2433
+2434
+ p.pu[LUMA_64x16].convert_p2s[ALIGNED] = PFX(filterPixelToShort_aligned_64x16_avx512);2435
+ p.pu[LUMA_64x32].convert_p2s[ALIGNED] = PFX(filterPixelToShort_aligned_64x32_avx512);2436
+ p.pu[LUMA_64x48].convert_p2s[ALIGNED] = PFX(filterPixelToShort_aligned_64x48_avx512);2437
+ p.pu[LUMA_64x64].convert_p2s[ALIGNED] = PFX(filterPixelToShort_aligned_64x64_avx512);2438
+ p.pu[LUMA_32x8].convert_p2s[ALIGNED] = PFX(filterPixelToShort_aligned_32x8_avx512);2439
+ p.pu[LUMA_32x16].convert_p2s[ALIGNED] = PFX(filterPixelToShort_aligned_32x16_avx512);2440
+ p.pu[LUMA_32x24].convert_p2s[ALIGNED] = PFX(filterPixelToShort_aligned_32x24_avx512);2441
+ p.pu[LUMA_32x32].convert_p2s[ALIGNED] = PFX(filterPixelToShort_aligned_32x32_avx512);2442
+ p.pu[LUMA_32x64].convert_p2s[ALIGNED] = PFX(filterPixelToShort_aligned_32x64_avx512);2443
+ p.pu[LUMA_48x64].convert_p2s[ALIGNED] = PFX(filterPixelToShort_aligned_48x64_avx512);2444
+2445
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x8].p2s[ALIGNED] = PFX(filterPixelToShort_aligned_32x8_avx512);2446
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x16].p2s[ALIGNED] = PFX(filterPixelToShort_aligned_32x16_avx512);2447
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x24].p2s[ALIGNED] = PFX(filterPixelToShort_aligned_32x24_avx512);2448
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x32].p2s[ALIGNED] = PFX(filterPixelToShort_aligned_32x32_avx512);2449
+2450
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].p2s[ALIGNED] = PFX(filterPixelToShort_aligned_32x16_avx512);2451
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x32].p2s[ALIGNED] = PFX(filterPixelToShort_aligned_32x32_avx512);2452
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x48].p2s[ALIGNED] = PFX(filterPixelToShort_aligned_32x48_avx512);2453
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x64].p2s[ALIGNED] = PFX(filterPixelToShort_aligned_32x64_avx512);2454
+2455
+ p.chroma[X265_CSP_I444].pu[LUMA_32x8].p2s[ALIGNED] = PFX(filterPixelToShort_aligned_32x8_avx512);2456
+ p.chroma[X265_CSP_I444].pu[LUMA_32x16].p2s[ALIGNED] = PFX(filterPixelToShort_aligned_32x16_avx512);2457
+ p.chroma[X265_CSP_I444].pu[LUMA_32x24].p2s[ALIGNED] = PFX(filterPixelToShort_aligned_32x24_avx512);2458
+ p.chroma[X265_CSP_I444].pu[LUMA_32x32].p2s[ALIGNED] = PFX(filterPixelToShort_aligned_32x32_avx512);2459
+ p.chroma[X265_CSP_I444].pu[LUMA_32x64].p2s[ALIGNED] = PFX(filterPixelToShort_aligned_32x64_avx512);2460
+ p.chroma[X265_CSP_I444].pu[LUMA_64x16].p2s[ALIGNED] = PFX(filterPixelToShort_aligned_64x16_avx512);2461
+ p.chroma[X265_CSP_I444].pu[LUMA_64x32].p2s[ALIGNED] = PFX(filterPixelToShort_aligned_64x32_avx512);2462
+ p.chroma[X265_CSP_I444].pu[LUMA_64x48].p2s[ALIGNED] = PFX(filterPixelToShort_aligned_64x48_avx512);2463
+ p.chroma[X265_CSP_I444].pu[LUMA_64x64].p2s[ALIGNED] = PFX(filterPixelToShort_aligned_64x64_avx512);2464
+2465
+ p.cu[BLOCK_64x64].sse_ss = (pixel_sse_ss_t)PFX(pixel_ssd_ss_64x64_avx512);2466
+ p.cu[BLOCK_32x32].sse_ss = (pixel_sse_ss_t)PFX(pixel_ssd_ss_32x32_avx512);2467
+ p.cu[BLOCK_16x16].sse_ss = (pixel_sse_ss_t)PFX(pixel_ssd_ss_16x16_avx512);2468
+ p.cu[BLOCK_32x32].ssd_s[NONALIGNED] = PFX(pixel_ssd_s_32_avx512);2469
+ p.cu[BLOCK_32x32].ssd_s[ALIGNED] = PFX(pixel_ssd_s_32_avx512);2470
+ p.cu[BLOCK_16x16].ssd_s[NONALIGNED] = PFX(pixel_ssd_s_16_avx512);2471
+ p.cu[BLOCK_16x16].ssd_s[ALIGNED] = PFX(pixel_ssd_s_aligned_16_avx512);2472
+ p.cu[BLOCK_32x32].copy_ss = PFX(blockcopy_ss_32x32_avx512);2473
+ p.chroma[X265_CSP_I420].cu[CHROMA_420_32x32].copy_ss = PFX(blockcopy_ss_32x32_avx512);2474
+ p.chroma[X265_CSP_I422].cu[CHROMA_422_32x64].copy_ss = PFX(blockcopy_ss_32x64_avx512);2475
+ p.cu[BLOCK_64x64].copy_ss = PFX(blockcopy_ss_64x64_avx512);2476
+2477
+ p.cu[BLOCK_32x32].calcresidual[NONALIGNED] = PFX(getResidual32_avx512);2478
+ p.cu[BLOCK_32x32].calcresidual[ALIGNED] = PFX(getResidual_aligned32_avx512);2479
+ p.cu[BLOCK_16x16].cpy2Dto1D_shl = PFX(cpy2Dto1D_shl_16_avx512);2480
+ p.cu[BLOCK_32x32].cpy2Dto1D_shl = PFX(cpy2Dto1D_shl_32_avx512);2481
+ p.cu[BLOCK_32x32].cpy1Dto2D_shl[NONALIGNED] = PFX(cpy1Dto2D_shl_32_avx512);2482
+ p.cu[BLOCK_32x32].cpy1Dto2D_shl[ALIGNED] = PFX(cpy1Dto2D_shl_aligned_32_avx512);2483
+ p.cu[BLOCK_16x16].cpy1Dto2D_shr = PFX(cpy1Dto2D_shr_16_avx512);2484
+ p.cu[BLOCK_32x32].cpy1Dto2D_shr = PFX(cpy1Dto2D_shr_32_avx512);2485
+2486
+ p.cu[BLOCK_16x16].cpy2Dto1D_shr = PFX(cpy2Dto1D_shr_16_avx512);2487
+ p.cu[BLOCK_32x32].cpy2Dto1D_shr = PFX(cpy2Dto1D_shr_32_avx512);2488
+2489
+ p.cu[BLOCK_32x32].copy_cnt = PFX(copy_cnt_32_avx512);2490
+ p.cu[BLOCK_16x16].copy_cnt = PFX(copy_cnt_16_avx512);2491
+2492
+ p.dequant_normal = PFX(dequant_normal_avx512);2493
+ p.dequant_scaling = PFX(dequant_scaling_avx512);2494
+ //i444 chroma_hpp2495
+ p.chroma[X265_CSP_I444].pu[LUMA_64x64].filter_hpp = PFX(interp_4tap_horiz_pp_64x64_avx512);2496
+ p.chroma[X265_CSP_I444].pu[LUMA_64x32].filter_hpp = PFX(interp_4tap_horiz_pp_64x32_avx512);2497
+ p.chroma[X265_CSP_I444].pu[LUMA_64x48].filter_hpp = PFX(interp_4tap_horiz_pp_64x48_avx512);2498
+ p.chroma[X265_CSP_I444].pu[LUMA_64x16].filter_hpp = PFX(interp_4tap_horiz_pp_64x16_avx512);2499
+ p.chroma[X265_CSP_I444].pu[LUMA_32x16].filter_hpp = PFX(interp_4tap_horiz_pp_32x16_avx512);2500
+ p.chroma[X265_CSP_I444].pu[LUMA_32x64].filter_hpp = PFX(interp_4tap_horiz_pp_32x64_avx512);2501
+ p.chroma[X265_CSP_I444].pu[LUMA_32x24].filter_hpp = PFX(interp_4tap_horiz_pp_32x24_avx512);2502
+ p.chroma[X265_CSP_I444].pu[LUMA_32x8].filter_hpp = PFX(interp_4tap_horiz_pp_32x8_avx512);2503
+ p.chroma[X265_CSP_I444].pu[LUMA_32x32].filter_hpp = PFX(interp_4tap_horiz_pp_32x32_avx512);2504
+ p.chroma[X265_CSP_I444].pu[LUMA_16x16].filter_hpp = PFX(interp_4tap_horiz_pp_16x16_avx512);2505
+ p.chroma[X265_CSP_I444].pu[LUMA_16x8].filter_hpp = PFX(interp_4tap_horiz_pp_16x8_avx512);2506
+ p.chroma[X265_CSP_I444].pu[LUMA_16x32].filter_hpp = PFX(interp_4tap_horiz_pp_16x32_avx512);2507
+ p.chroma[X265_CSP_I444].pu[LUMA_16x12].filter_hpp = PFX(interp_4tap_horiz_pp_16x12_avx512);2508
+ p.chroma[X265_CSP_I444].pu[LUMA_16x4].filter_hpp = PFX(interp_4tap_horiz_pp_16x4_avx512);2509
+ p.chroma[X265_CSP_I444].pu[LUMA_16x64].filter_hpp = PFX(interp_4tap_horiz_pp_16x64_avx512);2510
+ p.chroma[X265_CSP_I444].pu[LUMA_48x64].filter_hpp = PFX(interp_4tap_horiz_pp_48x64_avx512);2511
+2512
+ //i422 chroma_hpp2513
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x8].filter_hpp = PFX(interp_4tap_horiz_pp_16x8_avx512);2514
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x16].filter_hpp = PFX(interp_4tap_horiz_pp_16x16_avx512);2515
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x32].filter_hpp = PFX(interp_4tap_horiz_pp_16x32_avx512);2516
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x64].filter_hpp = PFX(interp_4tap_horiz_pp_16x64_avx512);2517
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x24].filter_hpp = PFX(interp_4tap_horiz_pp_16x24_avx512);2518
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].filter_hpp = PFX(interp_4tap_horiz_pp_32x16_avx512);2519
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x32].filter_hpp = PFX(interp_4tap_horiz_pp_32x32_avx512);2520
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x64].filter_hpp = PFX(interp_4tap_horiz_pp_32x64_avx512);2521
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x48].filter_hpp = PFX(interp_4tap_horiz_pp_32x48_avx512);2522
+2523
+ //i420 chroma_hpp2524
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x4].filter_hpp = PFX(interp_4tap_horiz_pp_16x4_avx512);2525
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x8].filter_hpp = PFX(interp_4tap_horiz_pp_16x8_avx512);2526
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x12].filter_hpp = PFX(interp_4tap_horiz_pp_16x12_avx512);2527
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x32].filter_hpp = PFX(interp_4tap_horiz_pp_16x32_avx512);2528
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x16].filter_hpp = PFX(interp_4tap_horiz_pp_16x16_avx512);2529
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x32].filter_hpp = PFX(interp_4tap_horiz_pp_32x32_avx512);2530
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x16].filter_hpp = PFX(interp_4tap_horiz_pp_32x16_avx512);2531
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x24].filter_hpp = PFX(interp_4tap_horiz_pp_32x24_avx512);2532
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x8].filter_hpp = PFX(interp_4tap_horiz_pp_32x8_avx512);2533
+2534
+ p.weight_pp = PFX(weight_pp_avx512);2535
+ p.weight_sp = PFX(weight_sp_avx512);2536
+2537
+ //i444 chroma_hps2538
+ p.chroma[X265_CSP_I444].pu[LUMA_64x64].filter_hps = PFX(interp_4tap_horiz_ps_64x64_avx512);2539
+ p.chroma[X265_CSP_I444].pu[LUMA_64x32].filter_hps = PFX(interp_4tap_horiz_ps_64x32_avx512);2540
+ p.chroma[X265_CSP_I444].pu[LUMA_64x48].filter_hps = PFX(interp_4tap_horiz_ps_64x48_avx512);2541
+ p.chroma[X265_CSP_I444].pu[LUMA_64x16].filter_hps = PFX(interp_4tap_horiz_ps_64x16_avx512);2542
+2543
+ p.chroma[X265_CSP_I444].pu[LUMA_32x32].filter_hps = PFX(interp_4tap_horiz_ps_32x32_avx512);2544
+ p.chroma[X265_CSP_I444].pu[LUMA_32x16].filter_hps = PFX(interp_4tap_horiz_ps_32x16_avx512);2545
+ p.chroma[X265_CSP_I444].pu[LUMA_32x64].filter_hps = PFX(interp_4tap_horiz_ps_32x64_avx512);2546
+ p.chroma[X265_CSP_I444].pu[LUMA_32x24].filter_hps = PFX(interp_4tap_horiz_ps_32x24_avx512);2547
+ p.chroma[X265_CSP_I444].pu[LUMA_32x8].filter_hps = PFX(interp_4tap_horiz_ps_32x8_avx512);2548
+2549
+ //i422 chroma_hps2550
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].filter_hps = PFX(interp_4tap_horiz_ps_32x16_avx512);2551
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x32].filter_hps = PFX(interp_4tap_horiz_ps_32x32_avx512);2552
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x64].filter_hps = PFX(interp_4tap_horiz_ps_32x64_avx512);2553
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x48].filter_hps = PFX(interp_4tap_horiz_ps_32x48_avx512);2554
+2555
+ //i420 chroma_hps2556
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x32].filter_hps = PFX(interp_4tap_horiz_ps_32x32_avx512);2557
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x16].filter_hps = PFX(interp_4tap_horiz_ps_32x16_avx512);2558
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x24].filter_hps = PFX(interp_4tap_horiz_ps_32x24_avx512);2559
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x8].filter_hps = PFX(interp_4tap_horiz_ps_32x8_avx512);2560
+2561
+ p.pu[LUMA_16x4].luma_hpp = PFX(interp_8tap_horiz_pp_16x4_avx512);2562
+ p.pu[LUMA_16x8].luma_hpp = PFX(interp_8tap_horiz_pp_16x8_avx512);2563
+ p.pu[LUMA_16x12].luma_hpp = PFX(interp_8tap_horiz_pp_16x12_avx512);2564
+ p.pu[LUMA_16x16].luma_hpp = PFX(interp_8tap_horiz_pp_16x16_avx512);2565
+ p.pu[LUMA_16x32].luma_hpp = PFX(interp_8tap_horiz_pp_16x32_avx512);2566
+ p.pu[LUMA_16x64].luma_hpp = PFX(interp_8tap_horiz_pp_16x64_avx512);2567
+ p.pu[LUMA_32x8].luma_hpp = PFX(interp_8tap_horiz_pp_32x8_avx512);2568
+ p.pu[LUMA_32x16].luma_hpp = PFX(interp_8tap_horiz_pp_32x16_avx512);2569
+ p.pu[LUMA_32x24].luma_hpp = PFX(interp_8tap_horiz_pp_32x24_avx512);2570
+ p.pu[LUMA_32x32].luma_hpp = PFX(interp_8tap_horiz_pp_32x32_avx512);2571
+ p.pu[LUMA_32x64].luma_hpp = PFX(interp_8tap_horiz_pp_32x64_avx512);2572
+ p.pu[LUMA_64x16].luma_hpp = PFX(interp_8tap_horiz_pp_64x16_avx512);2573
+ p.pu[LUMA_64x32].luma_hpp = PFX(interp_8tap_horiz_pp_64x32_avx512);2574
+ p.pu[LUMA_64x48].luma_hpp = PFX(interp_8tap_horiz_pp_64x48_avx512);2575
+ p.pu[LUMA_64x64].luma_hpp = PFX(interp_8tap_horiz_pp_64x64_avx512);2576
+ p.pu[LUMA_48x64].luma_hpp = PFX(interp_8tap_horiz_pp_48x64_avx512);2577
+ ASSIGN2(p.pu[LUMA_64x16].pixelavg_pp, pixel_avg_64x16_avx512);2578
+ ASSIGN2(p.pu[LUMA_64x32].pixelavg_pp, pixel_avg_64x32_avx512);2579
+ ASSIGN2(p.pu[LUMA_64x48].pixelavg_pp, pixel_avg_64x48_avx512);2580
+ ASSIGN2(p.pu[LUMA_64x64].pixelavg_pp, pixel_avg_64x64_avx512);2581
+ //luma hps2582
+ p.pu[LUMA_64x64].luma_hps = PFX(interp_8tap_horiz_ps_64x64_avx512);2583
+ p.pu[LUMA_64x48].luma_hps = PFX(interp_8tap_horiz_ps_64x48_avx512);2584
+ p.pu[LUMA_64x32].luma_hps = PFX(interp_8tap_horiz_ps_64x32_avx512);2585
+ p.pu[LUMA_64x16].luma_hps = PFX(interp_8tap_horiz_ps_64x16_avx512);2586
+2587
+ p.pu[LUMA_32x64].luma_hps = PFX(interp_8tap_horiz_ps_32x64_avx512);2588
+ p.pu[LUMA_32x32].luma_hps = PFX(interp_8tap_horiz_ps_32x32_avx512);2589
+ p.pu[LUMA_32x24].luma_hps = PFX(interp_8tap_horiz_ps_32x24_avx512);2590
+ p.pu[LUMA_32x16].luma_hps = PFX(interp_8tap_horiz_ps_32x16_avx512);2591
+ p.pu[LUMA_32x8].luma_hps = PFX(interp_8tap_horiz_ps_32x8_avx512);2592
+2593
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x32].filter_hps = PFX(interp_4tap_horiz_ps_16x32_avx512);2594
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x12].filter_hps = PFX(interp_4tap_horiz_ps_16x12_avx512);2595
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x8].filter_hps = PFX(interp_4tap_horiz_ps_16x8_avx512);2596
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x4].filter_hps = PFX(interp_4tap_horiz_ps_16x4_avx512);2597
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x16].filter_hps = PFX(interp_4tap_horiz_ps_16x16_avx512);2598
+2599
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x8].filter_hps = PFX(interp_4tap_horiz_ps_16x8_avx512);2600
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x16].filter_hps = PFX(interp_4tap_horiz_ps_16x16_avx512);2601
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x32].filter_hps = PFX(interp_4tap_horiz_ps_16x32_avx512);2602
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x64].filter_hps = PFX(interp_4tap_horiz_ps_16x64_avx512);2603
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x24].filter_hps = PFX(interp_4tap_horiz_ps_16x24_avx512);2604
+2605
+ p.chroma[X265_CSP_I444].pu[LUMA_16x16].filter_hps = PFX(interp_4tap_horiz_ps_16x16_avx512);2606
+ p.chroma[X265_CSP_I444].pu[LUMA_16x8].filter_hps = PFX(interp_4tap_horiz_ps_16x8_avx512);2607
+ p.chroma[X265_CSP_I444].pu[LUMA_16x32].filter_hps = PFX(interp_4tap_horiz_ps_16x32_avx512);2608
+ p.chroma[X265_CSP_I444].pu[LUMA_16x12].filter_hps = PFX(interp_4tap_horiz_ps_16x12_avx512);2609
+ p.chroma[X265_CSP_I444].pu[LUMA_16x4].filter_hps = PFX(interp_4tap_horiz_ps_16x4_avx512);2610
+ p.chroma[X265_CSP_I444].pu[LUMA_16x64].filter_hps = PFX(interp_4tap_horiz_ps_16x64_avx512);2611
+2612
+ p.pu[LUMA_16x8].luma_hps = PFX(interp_8tap_horiz_ps_16x8_avx512);2613
+ p.pu[LUMA_16x16].luma_hps = PFX(interp_8tap_horiz_ps_16x16_avx512);2614
+ p.pu[LUMA_16x12].luma_hps = PFX(interp_8tap_horiz_ps_16x12_avx512);2615
+ p.pu[LUMA_16x4].luma_hps = PFX(interp_8tap_horiz_ps_16x4_avx512);2616
+ p.pu[LUMA_16x32].luma_hps = PFX(interp_8tap_horiz_ps_16x32_avx512);2617
+ p.pu[LUMA_16x64].luma_hps = PFX(interp_8tap_horiz_ps_16x64_avx512);2618
+2619
+ p.pu[LUMA_48x64].luma_hps = PFX(interp_8tap_horiz_ps_48x64_avx512);2620
+ p.chroma[X265_CSP_I444].pu[LUMA_48x64].filter_hps = PFX(interp_4tap_horiz_ps_48x64_avx512);2621
+2622
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x4].filter_vpp = PFX(interp_4tap_vert_pp_16x4_avx512);2623
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x8].filter_vpp = PFX(interp_4tap_vert_pp_16x8_avx512);2624
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x12].filter_vpp = PFX(interp_4tap_vert_pp_16x12_avx512);2625
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x16].filter_vpp = PFX(interp_4tap_vert_pp_16x16_avx512);2626
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x32].filter_vpp = PFX(interp_4tap_vert_pp_16x32_avx512);2627
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x8].filter_vpp = PFX(interp_4tap_vert_pp_32x8_avx512);2628
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x16].filter_vpp = PFX(interp_4tap_vert_pp_32x16_avx512);2629
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x24].filter_vpp = PFX(interp_4tap_vert_pp_32x24_avx512);2630
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x32].filter_vpp = PFX(interp_4tap_vert_pp_32x32_avx512);2631
+2632
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_8x4].filter_vss = PFX(interp_4tap_vert_ss_8x4_avx512);2633
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_8x8].filter_vss = PFX(interp_4tap_vert_ss_8x8_avx512);2634
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_8x16].filter_vss = PFX(interp_4tap_vert_ss_8x16_avx512);2635
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_8x32].filter_vss = PFX(interp_4tap_vert_ss_8x32_avx512);2636
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x4].filter_vss = PFX(interp_4tap_vert_ss_16x4_avx512);2637
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x8].filter_vss = PFX(interp_4tap_vert_ss_16x8_avx512);2638
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x12].filter_vss = PFX(interp_4tap_vert_ss_16x12_avx512);2639
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x16].filter_vss = PFX(interp_4tap_vert_ss_16x16_avx512);2640
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x32].filter_vss = PFX(interp_4tap_vert_ss_16x32_avx512);2641
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_24x32].filter_vss = PFX(interp_4tap_vert_ss_24x32_avx512);2642
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x8].filter_vss = PFX(interp_4tap_vert_ss_32x8_avx512);2643
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x16].filter_vss = PFX(interp_4tap_vert_ss_32x16_avx512);2644
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x24].filter_vss = PFX(interp_4tap_vert_ss_32x24_avx512);2645
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x32].filter_vss = PFX(interp_4tap_vert_ss_32x32_avx512);2646
+2647
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x4].filter_vsp = PFX(interp_4tap_vert_sp_16x4_avx512);2648
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x8].filter_vsp = PFX(interp_4tap_vert_sp_16x8_avx512);2649
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x12].filter_vsp = PFX(interp_4tap_vert_sp_16x12_avx512);2650
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x16].filter_vsp = PFX(interp_4tap_vert_sp_16x16_avx512);2651
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x32].filter_vsp = PFX(interp_4tap_vert_sp_16x32_avx512);2652
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x8].filter_vsp = PFX(interp_4tap_vert_sp_32x8_avx512);2653
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x16].filter_vsp = PFX(interp_4tap_vert_sp_32x16_avx512);2654
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x24].filter_vsp = PFX(interp_4tap_vert_sp_32x24_avx512);2655
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x32].filter_vsp = PFX(interp_4tap_vert_sp_32x32_avx512);2656
+2657
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x8].filter_vpp = PFX(interp_4tap_vert_pp_16x8_avx512);2658
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x16].filter_vpp = PFX(interp_4tap_vert_pp_16x16_avx512);2659
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x24].filter_vpp = PFX(interp_4tap_vert_pp_16x24_avx512);2660
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x32].filter_vpp = PFX(interp_4tap_vert_pp_16x32_avx512);2661
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x64].filter_vpp = PFX(interp_4tap_vert_pp_16x64_avx512);2662
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x24].filter_vpp = PFX(interp_4tap_vert_pp_16x24_avx512);2663
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].filter_vpp = PFX(interp_4tap_vert_pp_32x16_avx512);2664
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x32].filter_vpp = PFX(interp_4tap_vert_pp_32x32_avx512);2665
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x48].filter_vpp = PFX(interp_4tap_vert_pp_32x48_avx512);2666
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x64].filter_vpp = PFX(interp_4tap_vert_pp_32x64_avx512);2667
+2668
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x4].filter_vss = PFX(interp_4tap_vert_ss_8x4_avx512);2669
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x8].filter_vss = PFX(interp_4tap_vert_ss_8x8_avx512);2670
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x12].filter_vss = PFX(interp_4tap_vert_ss_8x12_avx512);2671
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x16].filter_vss = PFX(interp_4tap_vert_ss_8x16_avx512);2672
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x32].filter_vss = PFX(interp_4tap_vert_ss_8x32_avx512);2673
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_8x64].filter_vss = PFX(interp_4tap_vert_ss_8x64_avx512);2674
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x8].filter_vss = PFX(interp_4tap_vert_ss_16x8_avx512);2675
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x16].filter_vss = PFX(interp_4tap_vert_ss_16x16_avx512);2676
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x24].filter_vss = PFX(interp_4tap_vert_ss_16x24_avx512);2677
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x32].filter_vss = PFX(interp_4tap_vert_ss_16x32_avx512);2678
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x64].filter_vss = PFX(interp_4tap_vert_ss_16x64_avx512);2679
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_24x64].filter_vss = PFX(interp_4tap_vert_ss_24x64_avx512);2680
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].filter_vss = PFX(interp_4tap_vert_ss_32x16_avx512);2681
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x32].filter_vss = PFX(interp_4tap_vert_ss_32x32_avx512);2682
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x48].filter_vss = PFX(interp_4tap_vert_ss_32x48_avx512);2683
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x64].filter_vss = PFX(interp_4tap_vert_ss_32x64_avx512);2684
+2685
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x8].filter_vsp = PFX(interp_4tap_vert_sp_16x8_avx512);2686
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x16].filter_vsp = PFX(interp_4tap_vert_sp_16x16_avx512);2687
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x24].filter_vsp = PFX(interp_4tap_vert_sp_16x24_avx512);2688
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x32].filter_vsp = PFX(interp_4tap_vert_sp_16x32_avx512);2689
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x64].filter_vsp = PFX(interp_4tap_vert_sp_16x64_avx512);2690
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].filter_vsp = PFX(interp_4tap_vert_sp_32x16_avx512);2691
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x32].filter_vsp = PFX(interp_4tap_vert_sp_32x32_avx512);2692
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x48].filter_vsp = PFX(interp_4tap_vert_sp_32x48_avx512);2693
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x64].filter_vsp = PFX(interp_4tap_vert_sp_32x64_avx512);2694
+2695
+ p.chroma[X265_CSP_I444].pu[LUMA_16x4].filter_vpp = PFX(interp_4tap_vert_pp_16x4_avx512);2696
+ p.chroma[X265_CSP_I444].pu[LUMA_16x8].filter_vpp = PFX(interp_4tap_vert_pp_16x8_avx512);2697
+ p.chroma[X265_CSP_I444].pu[LUMA_16x12].filter_vpp = PFX(interp_4tap_vert_pp_16x12_avx512);2698
+ p.chroma[X265_CSP_I444].pu[LUMA_16x16].filter_vpp = PFX(interp_4tap_vert_pp_16x16_avx512);2699
+ p.chroma[X265_CSP_I444].pu[LUMA_16x32].filter_vpp = PFX(interp_4tap_vert_pp_16x32_avx512);2700
+ p.chroma[X265_CSP_I444].pu[LUMA_16x64].filter_vpp = PFX(interp_4tap_vert_pp_16x64_avx512);2701
+ p.chroma[X265_CSP_I444].pu[LUMA_32x8].filter_vpp = PFX(interp_4tap_vert_pp_32x8_avx512);2702
+ p.chroma[X265_CSP_I444].pu[LUMA_32x16].filter_vpp = PFX(interp_4tap_vert_pp_32x16_avx512);2703
+ p.chroma[X265_CSP_I444].pu[LUMA_32x24].filter_vpp = PFX(interp_4tap_vert_pp_32x24_avx512);2704
+ p.chroma[X265_CSP_I444].pu[LUMA_32x32].filter_vpp = PFX(interp_4tap_vert_pp_32x32_avx512);2705
+ p.chroma[X265_CSP_I444].pu[LUMA_32x64].filter_vpp = PFX(interp_4tap_vert_pp_32x64_avx512);2706
+ p.chroma[X265_CSP_I444].pu[LUMA_48x64].filter_vpp = PFX(interp_4tap_vert_pp_48x64_avx512);2707
+ p.chroma[X265_CSP_I444].pu[LUMA_64x64].filter_vpp = PFX(interp_4tap_vert_pp_64x64_avx512);2708
+ p.chroma[X265_CSP_I444].pu[LUMA_64x48].filter_vpp = PFX(interp_4tap_vert_pp_64x48_avx512);2709
+ p.chroma[X265_CSP_I444].pu[LUMA_64x32].filter_vpp = PFX(interp_4tap_vert_pp_64x32_avx512);2710
+ p.chroma[X265_CSP_I444].pu[LUMA_64x16].filter_vpp = PFX(interp_4tap_vert_pp_64x16_avx512);2711
+2712
+ p.chroma[X265_CSP_I444].pu[LUMA_8x4].filter_vss = PFX(interp_4tap_vert_ss_8x4_avx512);2713
+ p.chroma[X265_CSP_I444].pu[LUMA_8x8].filter_vss = PFX(interp_4tap_vert_ss_8x8_avx512);2714
+ p.chroma[X265_CSP_I444].pu[LUMA_8x16].filter_vss = PFX(interp_4tap_vert_ss_8x16_avx512);2715
+ p.chroma[X265_CSP_I444].pu[LUMA_8x32].filter_vss = PFX(interp_4tap_vert_ss_8x32_avx512);2716
+ p.chroma[X265_CSP_I444].pu[LUMA_16x4].filter_vss = PFX(interp_4tap_vert_ss_16x4_avx512);2717
+ p.chroma[X265_CSP_I444].pu[LUMA_16x8].filter_vss = PFX(interp_4tap_vert_ss_16x8_avx512);2718
+ p.chroma[X265_CSP_I444].pu[LUMA_16x12].filter_vss = PFX(interp_4tap_vert_ss_16x12_avx512);2719
+ p.chroma[X265_CSP_I444].pu[LUMA_16x16].filter_vss = PFX(interp_4tap_vert_ss_16x16_avx512);2720
+ p.chroma[X265_CSP_I444].pu[LUMA_16x32].filter_vss = PFX(interp_4tap_vert_ss_16x32_avx512);2721
+ p.chroma[X265_CSP_I444].pu[LUMA_16x64].filter_vss = PFX(interp_4tap_vert_ss_16x64_avx512);2722
+ p.chroma[X265_CSP_I444].pu[LUMA_24x32].filter_vss = PFX(interp_4tap_vert_ss_24x32_avx512);2723
+ p.chroma[X265_CSP_I444].pu[LUMA_32x8].filter_vss = PFX(interp_4tap_vert_ss_32x8_avx512);2724
+ p.chroma[X265_CSP_I444].pu[LUMA_32x16].filter_vss = PFX(interp_4tap_vert_ss_32x16_avx512);2725
+ p.chroma[X265_CSP_I444].pu[LUMA_32x24].filter_vss = PFX(interp_4tap_vert_ss_32x24_avx512);2726
+ p.chroma[X265_CSP_I444].pu[LUMA_32x32].filter_vss = PFX(interp_4tap_vert_ss_32x32_avx512);2727
+ p.chroma[X265_CSP_I444].pu[LUMA_32x64].filter_vss = PFX(interp_4tap_vert_ss_32x64_avx512);2728
+ p.chroma[X265_CSP_I444].pu[LUMA_64x64].filter_vss = PFX(interp_4tap_vert_ss_64x64_avx512);2729
+ p.chroma[X265_CSP_I444].pu[LUMA_64x48].filter_vss = PFX(interp_4tap_vert_ss_64x48_avx512);2730
+ p.chroma[X265_CSP_I444].pu[LUMA_64x32].filter_vss = PFX(interp_4tap_vert_ss_64x32_avx512);2731
+ p.chroma[X265_CSP_I444].pu[LUMA_64x16].filter_vss = PFX(interp_4tap_vert_ss_64x16_avx512);2732
+ p.chroma[X265_CSP_I444].pu[LUMA_48x64].filter_vss = PFX(interp_4tap_vert_ss_48x64_avx512);2733
+2734
+ p.chroma[X265_CSP_I444].pu[LUMA_16x4].filter_vsp = PFX(interp_4tap_vert_sp_16x4_avx512);2735
+ p.chroma[X265_CSP_I444].pu[LUMA_16x8].filter_vsp = PFX(interp_4tap_vert_sp_16x8_avx512);2736
+ p.chroma[X265_CSP_I444].pu[LUMA_16x12].filter_vsp = PFX(interp_4tap_vert_sp_16x12_avx512);2737
+ p.chroma[X265_CSP_I444].pu[LUMA_16x16].filter_vsp = PFX(interp_4tap_vert_sp_16x16_avx512);2738
+ p.chroma[X265_CSP_I444].pu[LUMA_16x32].filter_vsp = PFX(interp_4tap_vert_sp_16x32_avx512);2739
+ p.chroma[X265_CSP_I444].pu[LUMA_16x64].filter_vsp = PFX(interp_4tap_vert_sp_16x64_avx512);2740
+ p.chroma[X265_CSP_I444].pu[LUMA_32x8].filter_vsp = PFX(interp_4tap_vert_sp_32x8_avx512);2741
+ p.chroma[X265_CSP_I444].pu[LUMA_32x16].filter_vsp = PFX(interp_4tap_vert_sp_32x16_avx512);2742
+ p.chroma[X265_CSP_I444].pu[LUMA_32x24].filter_vsp = PFX(interp_4tap_vert_sp_32x24_avx512);2743
+ p.chroma[X265_CSP_I444].pu[LUMA_32x32].filter_vsp = PFX(interp_4tap_vert_sp_32x32_avx512);2744
+ p.chroma[X265_CSP_I444].pu[LUMA_32x64].filter_vsp = PFX(interp_4tap_vert_sp_32x64_avx512);2745
+ p.chroma[X265_CSP_I444].pu[LUMA_48x64].filter_vsp = PFX(interp_4tap_vert_sp_48x64_avx512);2746
+ p.chroma[X265_CSP_I444].pu[LUMA_64x64].filter_vsp = PFX(interp_4tap_vert_sp_64x64_avx512);2747
+ p.chroma[X265_CSP_I444].pu[LUMA_64x48].filter_vsp = PFX(interp_4tap_vert_sp_64x48_avx512);2748
+ p.chroma[X265_CSP_I444].pu[LUMA_64x32].filter_vsp = PFX(interp_4tap_vert_sp_64x32_avx512);2749
+ p.chroma[X265_CSP_I444].pu[LUMA_64x16].filter_vsp = PFX(interp_4tap_vert_sp_64x16_avx512);2750
+2751
+ p.pu[LUMA_8x8].luma_vss = PFX(interp_8tap_vert_ss_8x8_avx512);2752
+ p.pu[LUMA_8x16].luma_vss = PFX(interp_8tap_vert_ss_8x16_avx512);2753
+ p.pu[LUMA_8x32].luma_vss = PFX(interp_8tap_vert_ss_8x32_avx512);2754
+ p.pu[LUMA_16x4].luma_vss = PFX(interp_8tap_vert_ss_16x4_avx512);2755
+ p.pu[LUMA_16x8].luma_vss = PFX(interp_8tap_vert_ss_16x8_avx512);2756
+ p.pu[LUMA_16x12].luma_vss = PFX(interp_8tap_vert_ss_16x12_avx512);2757
+ p.pu[LUMA_16x16].luma_vss = PFX(interp_8tap_vert_ss_16x16_avx512);2758
+ p.pu[LUMA_16x32].luma_vss = PFX(interp_8tap_vert_ss_16x32_avx512);2759
+ p.pu[LUMA_16x64].luma_vss = PFX(interp_8tap_vert_ss_16x64_avx512);2760
+ p.pu[LUMA_24x32].luma_vss = PFX(interp_8tap_vert_ss_24x32_avx512);2761
+ p.pu[LUMA_32x64].luma_vss = PFX(interp_8tap_vert_ss_32x64_avx512);2762
+ p.pu[LUMA_32x32].luma_vss = PFX(interp_8tap_vert_ss_32x32_avx512);2763
+ p.pu[LUMA_32x24].luma_vss = PFX(interp_8tap_vert_ss_32x24_avx512);2764
+ p.pu[LUMA_32x16].luma_vss = PFX(interp_8tap_vert_ss_32x16_avx512);2765
+ p.pu[LUMA_32x8].luma_vss = PFX(interp_8tap_vert_ss_32x8_avx512);2766
+ p.pu[LUMA_48x64].luma_vss = PFX(interp_8tap_vert_ss_48x64_avx512);2767
+ p.pu[LUMA_64x64].luma_vss = PFX(interp_8tap_vert_ss_64x64_avx512);2768
+ p.pu[LUMA_64x48].luma_vss = PFX(interp_8tap_vert_ss_64x48_avx512);2769
+ p.pu[LUMA_64x32].luma_vss = PFX(interp_8tap_vert_ss_64x32_avx512);2770
+ p.pu[LUMA_64x16].luma_vss = PFX(interp_8tap_vert_ss_64x16_avx512);2771
+2772
+ p.pu[LUMA_16x64].luma_vpp = PFX(interp_8tap_vert_pp_16x64_avx512);2773
+ p.pu[LUMA_16x32].luma_vpp = PFX(interp_8tap_vert_pp_16x32_avx512);2774
+ p.pu[LUMA_16x16].luma_vpp = PFX(interp_8tap_vert_pp_16x16_avx512);2775
+ p.pu[LUMA_16x8].luma_vpp = PFX(interp_8tap_vert_pp_16x8_avx512);2776
+ p.pu[LUMA_32x64].luma_vpp = PFX(interp_8tap_vert_pp_32x64_avx512);2777
+ p.pu[LUMA_32x32].luma_vpp = PFX(interp_8tap_vert_pp_32x32_avx512);2778
+ p.pu[LUMA_32x24].luma_vpp = PFX(interp_8tap_vert_pp_32x24_avx512);2779
+ p.pu[LUMA_32x16].luma_vpp = PFX(interp_8tap_vert_pp_32x16_avx512);2780
+ p.pu[LUMA_32x8].luma_vpp = PFX(interp_8tap_vert_pp_32x8_avx512);2781
+ p.pu[LUMA_48x64].luma_vpp = PFX(interp_8tap_vert_pp_48x64_avx512);2782
+ p.pu[LUMA_64x64].luma_vpp = PFX(interp_8tap_vert_pp_64x64_avx512);2783
+ p.pu[LUMA_64x48].luma_vpp = PFX(interp_8tap_vert_pp_64x48_avx512);2784
+ p.pu[LUMA_64x32].luma_vpp = PFX(interp_8tap_vert_pp_64x32_avx512);2785
+ p.pu[LUMA_64x16].luma_vpp = PFX(interp_8tap_vert_pp_64x16_avx512);2786
+ p.pu[LUMA_16x4].luma_vsp = PFX(interp_8tap_vert_sp_16x4_avx512);2787
+ p.pu[LUMA_16x8].luma_vsp = PFX(interp_8tap_vert_sp_16x8_avx512);2788
+ p.pu[LUMA_16x12].luma_vsp = PFX(interp_8tap_vert_sp_16x12_avx512);2789
+ p.pu[LUMA_16x16].luma_vsp = PFX(interp_8tap_vert_sp_16x16_avx512);2790
+ p.pu[LUMA_16x32].luma_vsp = PFX(interp_8tap_vert_sp_16x32_avx512);2791
+ p.pu[LUMA_16x64].luma_vsp = PFX(interp_8tap_vert_sp_16x64_avx512);2792
+ p.pu[LUMA_32x64].luma_vsp = PFX(interp_8tap_vert_sp_32x64_avx512);2793
+ p.pu[LUMA_32x32].luma_vsp = PFX(interp_8tap_vert_sp_32x32_avx512);2794
+ p.pu[LUMA_32x24].luma_vsp = PFX(interp_8tap_vert_sp_32x24_avx512);2795
+ p.pu[LUMA_32x16].luma_vsp = PFX(interp_8tap_vert_sp_32x16_avx512);2796
+ p.pu[LUMA_32x8].luma_vsp = PFX(interp_8tap_vert_sp_32x8_avx512);2797
+ p.pu[LUMA_48x64].luma_vsp = PFX(interp_8tap_vert_sp_48x64_avx512);2798
+ p.pu[LUMA_64x64].luma_vsp = PFX(interp_8tap_vert_sp_64x64_avx512);2799
+ p.pu[LUMA_64x48].luma_vsp = PFX(interp_8tap_vert_sp_64x48_avx512);2800
+ p.pu[LUMA_64x32].luma_vsp = PFX(interp_8tap_vert_sp_64x32_avx512);2801
+ p.pu[LUMA_64x16].luma_vsp = PFX(interp_8tap_vert_sp_64x16_avx512);2802
+2803
+ p.cu[BLOCK_8x8].dct = PFX(dct8_avx512);2804
+ /* TODO: Currently these kernels performance are similar to AVX2 version, we need a to improve them further to ebable2805
+ * it. Probably a Vtune analysis will help here.2806
+2807
+ * p.cu[BLOCK_16x16].dct = PFX(dct16_avx512);2808
+ * p.cu[BLOCK_32x32].dct = PFX(dct32_avx512); */2809
+2810
+ p.cu[BLOCK_8x8].idct = PFX(idct8_avx512);2811
+ p.cu[BLOCK_16x16].idct = PFX(idct16_avx512);2812
+ p.cu[BLOCK_32x32].idct = PFX(idct32_avx512);2813
+ p.quant = PFX(quant_avx512);2814
+ p.nquant = PFX(nquant_avx512);2815
+ p.denoiseDct = PFX(denoise_dct_avx512);2816
+2817
+ p.chroma[X265_CSP_I444].pu[LUMA_64x64].filter_vps = PFX(interp_4tap_vert_ps_64x64_avx512);2818
+ p.chroma[X265_CSP_I444].pu[LUMA_64x48].filter_vps = PFX(interp_4tap_vert_ps_64x48_avx512);2819
+ p.chroma[X265_CSP_I444].pu[LUMA_64x32].filter_vps = PFX(interp_4tap_vert_ps_64x32_avx512);2820
+ p.chroma[X265_CSP_I444].pu[LUMA_64x16].filter_vps = PFX(interp_4tap_vert_ps_64x16_avx512);2821
+2822
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x32].filter_vps = PFX(interp_4tap_vert_ps_32x32_avx512);2823
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x24].filter_vps = PFX(interp_4tap_vert_ps_32x24_avx512);2824
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x16].filter_vps = PFX(interp_4tap_vert_ps_32x16_avx512);2825
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x8].filter_vps = PFX(interp_4tap_vert_ps_32x8_avx512);2826
+2827
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x32].filter_vps = PFX(interp_4tap_vert_ps_32x32_avx512);2828
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].filter_vps = PFX(interp_4tap_vert_ps_32x16_avx512);2829
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x64].filter_vps = PFX(interp_4tap_vert_ps_32x64_avx512);2830
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x48].filter_vps = PFX(interp_4tap_vert_ps_32x48_avx512);2831
+2832
+ p.chroma[X265_CSP_I444].pu[LUMA_32x32].filter_vps = PFX(interp_4tap_vert_ps_32x32_avx512);2833
+ p.chroma[X265_CSP_I444].pu[LUMA_32x16].filter_vps = PFX(interp_4tap_vert_ps_32x16_avx512);2834
+ p.chroma[X265_CSP_I444].pu[LUMA_32x24].filter_vps = PFX(interp_4tap_vert_ps_32x24_avx512);2835
+ p.chroma[X265_CSP_I444].pu[LUMA_32x8].filter_vps = PFX(interp_4tap_vert_ps_32x8_avx512);2836
+ p.chroma[X265_CSP_I444].pu[LUMA_32x64].filter_vps = PFX(interp_4tap_vert_ps_32x64_avx512);2837
+2838
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x4].filter_vps = PFX(interp_4tap_vert_ps_16x4_avx512);2839
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x8].filter_vps = PFX(interp_4tap_vert_ps_16x8_avx512);2840
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x12].filter_vps = PFX(interp_4tap_vert_ps_16x12_avx512);2841
+ //p.chroma[X265_CSP_I420].pu[CHROMA_420_16x16].filter_vps = PFX(interp_4tap_vert_ps_16x16_avx512);2842
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_16x32].filter_vps = PFX(interp_4tap_vert_ps_16x32_avx512);2843
+2844
+ /*p.chroma[X265_CSP_I422].pu[CHROMA_422_16x32].filter_vps = PFX(interp_4tap_vert_ps_16x32_avx512);2845
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x16].filter_vps = PFX(interp_4tap_vert_ps_16x16_avx512);2846
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x8].filter_vps = PFX(interp_4tap_vert_ps_16x8_avx512);2847
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x64].filter_vps = PFX(interp_4tap_vert_ps_16x64_avx512);2848
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_16x24].filter_vps = PFX(interp_4tap_vert_ps_16x24_avx512);*/2849
+2850
+ //p.chroma[X265_CSP_I444].pu[LUMA_16x16].filter_vps = PFX(interp_4tap_vert_ps_16x16_avx512);2851
+ p.chroma[X265_CSP_I444].pu[LUMA_16x8].filter_vps = PFX(interp_4tap_vert_ps_16x8_avx512);2852
+ p.chroma[X265_CSP_I444].pu[LUMA_16x32].filter_vps = PFX(interp_4tap_vert_ps_16x32_avx512);2853
+ //p.chroma[X265_CSP_I444].pu[LUMA_16x12].filter_vps = PFX(interp_4tap_vert_ps_16x12_avx512);2854
+ p.chroma[X265_CSP_I444].pu[LUMA_16x4].filter_vps = PFX(interp_4tap_vert_ps_16x4_avx512);2855
+ p.chroma[X265_CSP_I444].pu[LUMA_16x64].filter_vps = PFX(interp_4tap_vert_ps_16x64_avx512);2856
+ p.cu[BLOCK_16x16].psy_cost_pp = PFX(psyCost_pp_16x16_avx512);2857
+ p.cu[BLOCK_32x32].psy_cost_pp = PFX(psyCost_pp_32x32_avx512);2858
+ p.cu[BLOCK_64x64].psy_cost_pp = PFX(psyCost_pp_64x64_avx512);2859
+2860
+ p.chroma[X265_CSP_I444].pu[LUMA_48x64].filter_vps = PFX(interp_4tap_vert_ps_48x64_avx512);2861
+2862
+ p.pu[LUMA_64x16].luma_vps = PFX(interp_8tap_vert_ps_64x16_avx512);2863
+ p.pu[LUMA_64x32].luma_vps = PFX(interp_8tap_vert_ps_64x32_avx512);2864
+ p.pu[LUMA_64x48].luma_vps = PFX(interp_8tap_vert_ps_64x48_avx512);2865
+ p.pu[LUMA_64x64].luma_vps = PFX(interp_8tap_vert_ps_64x64_avx512);2866
+2867
+ p.pu[LUMA_32x8].luma_vps = PFX(interp_8tap_vert_ps_32x8_avx512);2868
+ p.pu[LUMA_32x16].luma_vps = PFX(interp_8tap_vert_ps_32x16_avx512);2869
+ p.pu[LUMA_32x32].luma_vps = PFX(interp_8tap_vert_ps_32x32_avx512);2870
+ p.pu[LUMA_32x24].luma_vps = PFX(interp_8tap_vert_ps_32x24_avx512);2871
+ p.pu[LUMA_32x64].luma_vps = PFX(interp_8tap_vert_ps_32x64_avx512);2872
+2873
+ p.pu[LUMA_16x8].luma_vps = PFX(interp_8tap_vert_ps_16x8_avx512);2874
+ p.pu[LUMA_16x16].luma_vps = PFX(interp_8tap_vert_ps_16x16_avx512);2875
+ p.pu[LUMA_16x32].luma_vps = PFX(interp_8tap_vert_ps_16x32_avx512);2876
+ //p.pu[LUMA_16x64].luma_vps = PFX(interp_8tap_vert_ps_16x64_avx512);2877
+ p.pu[LUMA_48x64].luma_vps = PFX(interp_8tap_vert_ps_48x64_avx512);2878
+2879
+ p.pu[LUMA_64x64].luma_hvpp = interp_8tap_hv_pp_cpu<LUMA_64x64>;2880
+ p.pu[LUMA_64x48].luma_hvpp = interp_8tap_hv_pp_cpu<LUMA_64x48>;2881
+ p.pu[LUMA_64x32].luma_hvpp = interp_8tap_hv_pp_cpu<LUMA_64x32>;2882
+ p.pu[LUMA_64x16].luma_hvpp = interp_8tap_hv_pp_cpu<LUMA_64x16>;2883
+ p.pu[LUMA_32x8].luma_hvpp = interp_8tap_hv_pp_cpu<LUMA_32x8>;2884
+ p.pu[LUMA_32x16].luma_hvpp = interp_8tap_hv_pp_cpu<LUMA_32x16>;2885
+ p.pu[LUMA_32x32].luma_hvpp = interp_8tap_hv_pp_cpu<LUMA_32x32>;2886
+ p.pu[LUMA_32x24].luma_hvpp = interp_8tap_hv_pp_cpu<LUMA_32x24>;2887
+ p.pu[LUMA_32x64].luma_hvpp = interp_8tap_hv_pp_cpu<LUMA_32x64>;2888
+ p.pu[LUMA_16x4].luma_hvpp = interp_8tap_hv_pp_cpu<LUMA_16x4>;2889
+ p.pu[LUMA_16x8].luma_hvpp = interp_8tap_hv_pp_cpu<LUMA_16x8>;2890
+ p.pu[LUMA_16x12].luma_hvpp = interp_8tap_hv_pp_cpu<LUMA_16x12>;2891
+ p.pu[LUMA_16x16].luma_hvpp = interp_8tap_hv_pp_cpu<LUMA_16x16>;2892
+ p.pu[LUMA_16x32].luma_hvpp = interp_8tap_hv_pp_cpu<LUMA_16x32>;2893
+ p.pu[LUMA_16x64].luma_hvpp = interp_8tap_hv_pp_cpu<LUMA_16x64>;2894
+ p.pu[LUMA_48x64].luma_hvpp = interp_8tap_hv_pp_cpu<LUMA_48x64>;2895
+2896
+ p.cu[BLOCK_4x4].nonPsyRdoQuant = PFX(nonPsyRdoQuant4_avx512);2897
+ p.cu[BLOCK_8x8].nonPsyRdoQuant = PFX(nonPsyRdoQuant8_avx512);2898
+ p.cu[BLOCK_16x16].nonPsyRdoQuant = PFX(nonPsyRdoQuant16_avx512);2899
+ p.cu[BLOCK_32x32].nonPsyRdoQuant = PFX(nonPsyRdoQuant32_avx512);2900
+ p.cu[BLOCK_4x4].psyRdoQuant = PFX(psyRdoQuant4_avx512);2901
+ p.cu[BLOCK_8x8].psyRdoQuant = PFX(psyRdoQuant8_avx512);2902
+ p.cu[BLOCK_16x16].psyRdoQuant = PFX(psyRdoQuant16_avx512);2903
+ p.cu[BLOCK_32x32].psyRdoQuant = PFX(psyRdoQuant32_avx512);2904
+ p.pu[LUMA_32x8].satd = PFX(pixel_satd_32x8_avx512);2905
+ p.pu[LUMA_32x16].satd = PFX(pixel_satd_32x16_avx512);2906
+ p.pu[LUMA_32x24].satd = PFX(pixel_satd_32x24_avx512);2907
+ p.pu[LUMA_32x32].satd = PFX(pixel_satd_32x32_avx512);2908
+ p.pu[LUMA_32x64].satd = PFX(pixel_satd_32x64_avx512);2909
+ p.pu[LUMA_64x16].satd = PFX(pixel_satd_64x16_avx512);2910
+ p.pu[LUMA_64x32].satd = PFX(pixel_satd_64x32_avx512);2911
+ p.pu[LUMA_64x48].satd = PFX(pixel_satd_64x48_avx512);2912
+ p.pu[LUMA_64x64].satd = PFX(pixel_satd_64x64_avx512);2913
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x32].satd = PFX(pixel_satd_32x32_avx512);2914
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x16].satd = PFX(pixel_satd_32x16_avx512);2915
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x24].satd = PFX(pixel_satd_32x24_avx512);2916
+ p.chroma[X265_CSP_I420].pu[CHROMA_420_32x8].satd = PFX(pixel_satd_32x8_avx512);2917
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x64].satd = PFX(pixel_satd_32x64_avx512);2918
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x48].satd = PFX(pixel_satd_32x48_avx512);2919
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x32].satd = PFX(pixel_satd_32x32_avx512);2920
+ p.chroma[X265_CSP_I422].pu[CHROMA_422_32x16].satd = PFX(pixel_satd_32x16_avx512);2921
+ p.planecopy_sp_shl = PFX(upShift_16_avx512);2922
+ p.cu[BLOCK_16x16].count_nonzero = PFX(count_nonzero_16x16_avx512);2923
+ p.cu[BLOCK_32x32].count_nonzero = PFX(count_nonzero_32x32_avx512);2924
2925
}2926
#endif2927
2928
// CPU dispatcher function2929
void PFX(intel_cpu_indicator_init)(void)2930
{2931
- uint32_t cpu = x265::cpu_detect();2932
+ uint32_t cpu = x265::cpu_detect(false);2933
2934
if (cpu & X265_CPU_AVX)2935
__intel_cpu_indicator = 0x20000;2936
x265_2.7.tar.gz/source/common/x86/blockcopy8.asm -> x265_2.9.tar.gz/source/common/x86/blockcopy8.asm
Changed
1157
1
2
%include "x86inc.asm"3
%include "x86util.asm"4
5
-SECTION_RODATA 326
+SECTION_RODATA 647
+8
+ALIGN 649
+const shuf1_avx512, dq 0, 2, 4, 6, 1, 3, 5, 710
11
cextern pb_412
cextern pb_113
14
BLOCKCOPY_PP_W64_H4_avx 64, 4815
BLOCKCOPY_PP_W64_H4_avx 64, 6416
17
+;----------------------------------------------------------------------------------------------18
+; blockcopy_pp avx512 code start19
+;----------------------------------------------------------------------------------------------20
+%macro PROCESS_BLOCKCOPY_PP_64X4_avx512 021
+movu m0, [r2]22
+movu m1, [r2 + r3]23
+movu m2, [r2 + 2 * r3]24
+movu m3, [r2 + r4]25
+26
+movu [r0] , m027
+movu [r0 + r1] , m128
+movu [r0 + 2 * r1] , m229
+movu [r0 + r5] , m330
+%endmacro31
+32
+%macro PROCESS_BLOCKCOPY_PP_32X4_avx512 033
+movu ym0, [r2]34
+vinserti32x8 m0, [r2 + r3], 135
+movu ym1, [r2 + 2 * r3]36
+vinserti32x8 m1, [r2 + r4], 137
+38
+movu [r0] , ym039
+vextracti32x8 [r0 + r1] , m0, 140
+movu [r0 + 2 * r1] , ym141
+vextracti32x8 [r0 + r5] , m1, 142
+%endmacro43
+44
+;----------------------------------------------------------------------------------------------45
+; void blockcopy_pp_64x%1(pixel* dst, intptr_t dstStride, const pixel* src, intptr_t srcStride)46
+;----------------------------------------------------------------------------------------------47
+%macro BLOCKCOPY_PP_W64_H4_avx512 148
+INIT_ZMM avx51249
+cglobal blockcopy_pp_64x%1, 4, 6, 450
+lea r4, [3 * r3]51
+lea r5, [3 * r1]52
+53
+%rep %1/4 - 154
+PROCESS_BLOCKCOPY_PP_64X4_avx51255
+lea r2, [r2 + 4 * r3]56
+lea r0, [r0 + 4 * r1] 57
+%endrep58
+59
+PROCESS_BLOCKCOPY_PP_64X4_avx51260
+RET61
+%endmacro62
+63
+BLOCKCOPY_PP_W64_H4_avx512 1664
+BLOCKCOPY_PP_W64_H4_avx512 3265
+BLOCKCOPY_PP_W64_H4_avx512 4866
+BLOCKCOPY_PP_W64_H4_avx512 6467
+68
+%macro BLOCKCOPY_PP_W32_H4_avx512 169
+INIT_ZMM avx51270
+cglobal blockcopy_pp_32x%1, 4, 6, 271
+ lea r4, [3 * r3]72
+ lea r5, [3 * r1]73
+74
+%rep %1/4 - 175
+ PROCESS_BLOCKCOPY_PP_32X4_avx51276
+ lea r2, [r2 + 4 * r3]77
+ lea r0, [r0 + 4 * r1] 78
+%endrep79
+ PROCESS_BLOCKCOPY_PP_32X4_avx51280
+ RET81
+%endmacro82
+83
+BLOCKCOPY_PP_W32_H4_avx512 884
+BLOCKCOPY_PP_W32_H4_avx512 1685
+BLOCKCOPY_PP_W32_H4_avx512 2486
+BLOCKCOPY_PP_W32_H4_avx512 3287
+BLOCKCOPY_PP_W32_H4_avx512 4888
+BLOCKCOPY_PP_W32_H4_avx512 6489
+;----------------------------------------------------------------------------------------------90
+; blockcopy_pp avx512 code end91
+;----------------------------------------------------------------------------------------------92
+93
;-----------------------------------------------------------------------------94
; void blockcopy_sp_2x4(pixel* dst, intptr_t dstStride, const int16_t* src, intptr_t srcStride)95
;-----------------------------------------------------------------------------96
97
98
BLOCKCOPY_SP_W64_H4_avx2 64, 6499
100
+%macro PROCESS_BLOCKCOPY_SP_64x4_AVX512 0101
+ movu m0, [r2]102
+ movu m1, [r2 + 64]103
+ movu m2, [r2 + r3]104
+ movu m3, [r2 + r3 + 64]105
+106
+ packuswb m0, m1107
+ packuswb m2, m3108
+ vpermq m0, m4, m0109
+ vpermq m2, m4, m2110
+ movu [r0], m0111
+ movu [r0 + r1], m2112
+113
+ movu m0, [r2 + 2 * r3]114
+ movu m1, [r2 + 2 * r3 + 64]115
+ movu m2, [r2 + r4]116
+ movu m3, [r2 + r4 + 64]117
+118
+ packuswb m0, m1119
+ packuswb m2, m3120
+ vpermq m0, m4, m0121
+ vpermq m2, m4, m2122
+ movu [r0 + 2 * r1], m0123
+ movu [r0 + r5], m2124
+%endmacro125
+126
+%macro PROCESS_BLOCKCOPY_SP_32x4_AVX512 0127
+ movu m0, [r2]128
+ movu m1, [r2 + r3]129
+ movu m2, [r2 + 2 * r3]130
+ movu m3, [r2 + r4]131
+132
+ packuswb m0, m1133
+ packuswb m2, m3134
+ vpermq m0, m4, m0135
+ vpermq m2, m4, m2136
+ movu [r0], ym0137
+ vextracti32x8 [r0 + r1], m0, 1138
+ movu [r0 + 2 * r1], ym2139
+ vextracti32x8 [r0 + r5], m2, 1140
+%endmacro141
+142
+;-----------------------------------------------------------------------------143
+; void blockcopy_sp_%1x%2(pixel* dst, intptr_t dstStride, const int16_t* src, intptr_t srcStride)144
+;-----------------------------------------------------------------------------145
+INIT_ZMM avx512146
+cglobal blockcopy_sp_64x64, 4, 6, 5147
+ mova m4, [shuf1_avx512]148
+ add r3, r3149
+ lea r4, [3 * r3]150
+ lea r5, [3 * r1]151
+152
+%rep 15153
+ PROCESS_BLOCKCOPY_SP_64x4_AVX512154
+ lea r0, [r0 + 4 * r1]155
+ lea r2, [r2 + 4 * r3]156
+%endrep157
+ PROCESS_BLOCKCOPY_SP_64x4_AVX512158
+ RET159
+160
+%macro BLOCKCOPY_SP_32xN_AVX512 1161
+INIT_ZMM avx512162
+cglobal blockcopy_sp_32x%1, 4, 6, 5163
+ mova m4, [shuf1_avx512]164
+ add r3, r3165
+ lea r4, [3 * r3]166
+ lea r5, [3 * r1]167
+168
+%rep %1/4 - 1169
+ PROCESS_BLOCKCOPY_SP_32x4_AVX512170
+ lea r0, [r0 + 4 * r1]171
+ lea r2, [r2 + 4 * r3]172
+%endrep173
+ PROCESS_BLOCKCOPY_SP_32x4_AVX512174
+ RET175
+%endmacro176
+177
+BLOCKCOPY_SP_32xN_AVX512 32178
+BLOCKCOPY_SP_32xN_AVX512 64179
+180
;-----------------------------------------------------------------------------181
; void blockfill_s_4x4(int16_t* dst, intptr_t dstride, int16_t val)182
;-----------------------------------------------------------------------------183
184
movu [r0 + r3 + 32], m0185
RET186
187
+;--------------------------------------------------------------------188
+; void blockfill_s_32x32(int16_t* dst, intptr_t dstride, int16_t val)189
+;--------------------------------------------------------------------190
+INIT_ZMM avx512191
+cglobal blockfill_s_32x32, 3, 4, 1192
+add r1, r1193
+lea r3, [3 * r1]194
+movd xm0, r2d195
+vpbroadcastw m0, xm0196
+197
+%rep 8198
+movu [r0], m0199
+movu [r0 + r1], m0200
+movu [r0 + 2 * r1], m0201
+movu [r0 + r3], m0202
+lea r0, [r0 + 4 * r1]203
+%endrep204
+RET205
+206
+;--------------------------------------------------------------------207
+; void blockfill_s_aligned_32x32(int16_t* dst, intptr_t dstride, int16_t val)208
+;--------------------------------------------------------------------209
+INIT_ZMM avx512210
+cglobal blockfill_s_aligned_32x32, 3, 4, 1211
+add r1, r1212
+lea r3, [3 * r1]213
+movd xm0, r2d214
+vpbroadcastw m0, xm0215
+216
+%rep 8217
+mova [r0], m0218
+mova [r0 + r1], m0219
+mova [r0 + 2 * r1], m0220
+mova [r0 + r3], m0221
+lea r0, [r0 + 4 * r1]222
+%endrep223
+RET224
;-----------------------------------------------------------------------------225
; void blockcopy_ps_2x4(int16_t* dst, intptr_t dstStride, const pixel* src, intptr_t srcStride);226
;-----------------------------------------------------------------------------227
228
BLOCKCOPY_PS_W32_H4_avx2 32, 32229
BLOCKCOPY_PS_W32_H4_avx2 32, 64230
231
+%macro PROCESS_BLOCKCOPY_PS_32x8_AVX512 0232
+ pmovzxbw m0, [r2]233
+ pmovzxbw m1, [r2 + r3]234
+ pmovzxbw m2, [r2 + r3 * 2]235
+ pmovzxbw m3, [r2 + r4]236
+237
+ movu [r0], m0238
+ movu [r0 + r1], m1239
+ movu [r0 + r1 * 2], m2240
+ movu [r0 + r5], m3241
+242
+ lea r0, [r0 + 4 * r1]243
+ lea r2, [r2 + 4 * r3]244
+245
+ pmovzxbw m0, [r2]246
+ pmovzxbw m1, [r2 + r3]247
+ pmovzxbw m2, [r2 + r3 * 2]248
+ pmovzxbw m3, [r2 + r4]249
+250
+ movu [r0], m0251
+ movu [r0 + r1], m1252
+ movu [r0 + r1 * 2], m2253
+ movu [r0 + r5], m3254
+%endmacro255
+256
+INIT_ZMM avx512257
+cglobal blockcopy_ps_32x32, 4, 6, 4258
+ add r1, r1259
+ lea r4, [3 * r3]260
+ lea r5, [3 * r1]261
+262
+ PROCESS_BLOCKCOPY_PS_32x8_AVX512263
+ lea r0, [r0 + 4 * r1]264
+ lea r2, [r2 + 4 * r3]265
+ PROCESS_BLOCKCOPY_PS_32x8_AVX512266
+ lea r0, [r0 + 4 * r1]267
+ lea r2, [r2 + 4 * r3]268
+ PROCESS_BLOCKCOPY_PS_32x8_AVX512269
+ lea r0, [r0 + 4 * r1]270
+ lea r2, [r2 + 4 * r3]271
+ PROCESS_BLOCKCOPY_PS_32x8_AVX512272
+ RET273
+274
+INIT_ZMM avx512275
+cglobal blockcopy_ps_32x64, 4, 6, 4276
+ add r1, r1277
+ lea r4, [3 * r3]278
+ lea r5, [3 * r1]279
+280
+ PROCESS_BLOCKCOPY_PS_32x8_AVX512281
+ lea r0, [r0 + 4 * r1]282
+ lea r2, [r2 + 4 * r3]283
+ PROCESS_BLOCKCOPY_PS_32x8_AVX512284
+ lea r0, [r0 + 4 * r1]285
+ lea r2, [r2 + 4 * r3]286
+ PROCESS_BLOCKCOPY_PS_32x8_AVX512287
+ lea r0, [r0 + 4 * r1]288
+ lea r2, [r2 + 4 * r3]289
+ PROCESS_BLOCKCOPY_PS_32x8_AVX512290
+ lea r0, [r0 + 4 * r1]291
+ lea r2, [r2 + 4 * r3]292
+ PROCESS_BLOCKCOPY_PS_32x8_AVX512293
+ lea r0, [r0 + 4 * r1]294
+ lea r2, [r2 + 4 * r3]295
+ PROCESS_BLOCKCOPY_PS_32x8_AVX512296
+ lea r0, [r0 + 4 * r1]297
+ lea r2, [r2 + 4 * r3]298
+ PROCESS_BLOCKCOPY_PS_32x8_AVX512299
+ lea r0, [r0 + 4 * r1]300
+ lea r2, [r2 + 4 * r3]301
+ PROCESS_BLOCKCOPY_PS_32x8_AVX512302
+ RET303
+304
;-----------------------------------------------------------------------------305
; void blockcopy_ps_%1x%2(int16_t* dst, intptr_t dstStride, const pixel* src, intptr_t srcStride);306
;-----------------------------------------------------------------------------307
308
jnz .loop309
RET310
311
+%macro PROCESS_BLOCKCOPY_PS_64x8_AVX512 0312
+ pmovzxbw m0, [r2]313
+ pmovzxbw m1, [r2 + 32]314
+ pmovzxbw m2, [r2 + r3]315
+ pmovzxbw m3, [r2 + r3 + 32]316
+ movu [r0], m0317
+ movu [r0 + 64], m1318
+ movu [r0 + r1], m2319
+ movu [r0 + r1 + 64], m3320
+321
+ pmovzxbw m0, [r2 + r3 * 2]322
+ pmovzxbw m1, [r2 + r3 * 2 + 32]323
+ pmovzxbw m2, [r2 + r4]324
+ pmovzxbw m3, [r2 + r4 + 32]325
+ movu [r0 + r1 * 2], m0326
+ movu [r0 + r1 * 2 + 64], m1327
+ movu [r0 + r5], m2328
+ movu [r0 + r5 + 64], m3329
+330
+ lea r0, [r0 + 4 * r1]331
+ lea r2, [r2 + 4 * r3]332
+333
+ pmovzxbw m0, [r2]334
+ pmovzxbw m1, [r2 + 32]335
+ pmovzxbw m2, [r2 + r3]336
+ pmovzxbw m3, [r2 + r3 + 32]337
+ movu [r0], m0338
+ movu [r0 + 64], m1339
+ movu [r0 + r1], m2340
+ movu [r0 + r1 + 64], m3341
+342
+ pmovzxbw m0, [r2 + r3 * 2]343
+ pmovzxbw m1, [r2 + r3 * 2 + 32]344
+ pmovzxbw m2, [r2 + r4]345
+ pmovzxbw m3, [r2 + r4 + 32]346
+ movu [r0 + r1 * 2], m0347
+ movu [r0 + r1 * 2 + 64], m1348
+ movu [r0 + r5], m2349
+ movu [r0 + r5 + 64], m3350
+%endmacro351
+;-----------------------------------------------------------------------------352
+; void blockcopy_ps_%1x%2(int16_t* dst, intptr_t dstStride, const pixel* src, intptr_t srcStride);353
+;-----------------------------------------------------------------------------354
+INIT_ZMM avx512355
+cglobal blockcopy_ps_64x64, 4, 6, 4356
+ add r1, r1357
+ lea r4, [3 * r3]358
+ lea r5, [3 * r1]359
+360
+ PROCESS_BLOCKCOPY_PS_64x8_AVX512361
+ lea r0, [r0 + 4 * r1]362
+ lea r2, [r2 + 4 * r3]363
+ PROCESS_BLOCKCOPY_PS_64x8_AVX512364
+ lea r0, [r0 + 4 * r1]365
+ lea r2, [r2 + 4 * r3]366
+ PROCESS_BLOCKCOPY_PS_64x8_AVX512367
+ lea r0, [r0 + 4 * r1]368
+ lea r2, [r2 + 4 * r3]369
+ PROCESS_BLOCKCOPY_PS_64x8_AVX512370
+ lea r0, [r0 + 4 * r1]371
+ lea r2, [r2 + 4 * r3]372
+ PROCESS_BLOCKCOPY_PS_64x8_AVX512373
+ lea r0, [r0 + 4 * r1]374
+ lea r2, [r2 + 4 * r3]375
+ PROCESS_BLOCKCOPY_PS_64x8_AVX512376
+ lea r0, [r0 + 4 * r1]377
+ lea r2, [r2 + 4 * r3]378
+ PROCESS_BLOCKCOPY_PS_64x8_AVX512379
+ lea r0, [r0 + 4 * r1]380
+ lea r2, [r2 + 4 * r3]381
+ PROCESS_BLOCKCOPY_PS_64x8_AVX512382
+ RET383
+384
;-----------------------------------------------------------------------------385
; void blockcopy_ss_2x4(int16_t* dst, intptr_t dstStride, const int16_t* src, intptr_t srcStride)386
;-----------------------------------------------------------------------------387
388
BLOCKCOPY_SS_W32_H4_avx 32, 48389
BLOCKCOPY_SS_W32_H4_avx 32, 64390
391
+%macro PROCESS_BLOCKCOPY_SS_W32_H8_avx512 0392
+ movu m0, [r2]393
+ movu m1, [r2 + r3]394
+ movu m2, [r2 + 2 * r3]395
+ movu m3, [r2 + r6]396
+ lea r2, [r2 + 4 * r3]397
+398
+ movu [r0], m0399
+ movu [r0 + r1], m1400
+ movu [r0 + 2 * r1], m2401
+ movu [r0 + r5], m3402
+ lea r0, [r0 + 4 * r1]403
+404
+ movu m0, [r2]405
+ movu m1, [r2 + r3]406
+ movu m2, [r2 + 2 * r3]407
+ movu m3, [r2 + r6]408
+ lea r2, [r2 + 4 * r3]409
+410
+ movu [r0], m0411
+ movu [r0 + r1], m1412
+ movu [r0 + 2 * r1], m2413
+ movu [r0 + r5], m3414
+ lea r0, [r0 + 4 * r1]415
+%endmacro416
+417
+%macro PROCESS_BLOCKCOPY_SS_W32_H8_LAST_avx512 0418
+ movu m0, [r2]419
+ movu m1, [r2 + r3]420
+ movu m2, [r2 + 2 * r3]421
+ movu m3, [r2 + r6]422
+ lea r2, [r2 + 4 * r3]423
+424
+ movu [r0], m0425
+ movu [r0 + r1], m1426
+ movu [r0 + 2 * r1], m2427
+ movu [r0 + r5], m3428
+ lea r0, [r0 + 4 * r1]429
+430
+ movu m0, [r2]431
+ movu m1, [r2 + r3]432
+ movu m2, [r2 + 2 * r3]433
+ movu m3, [r2 + r6]434
+435
+ movu [r0], m0436
+ movu [r0 + r1], m1437
+ movu [r0 + 2 * r1], m2438
+ movu [r0 + r5], m3439
+%endmacro440
+441
+;-----------------------------------------------------------------------------442
+; void blockcopy_ss_%1x%2(int16_t* dst, intptr_t dstStride, const int16_t* src, intptr_t srcStride)443
+;-----------------------------------------------------------------------------444
+INIT_ZMM avx512445
+cglobal blockcopy_ss_32x8, 4, 7, 4446
+447
+ add r1, r1448
+ add r3, r3449
+ lea r5, [3 * r1]450
+ lea r6, [3 * r3]451
+452
+ PROCESS_BLOCKCOPY_SS_W32_H8_LAST_avx512453
+ RET454
+455
+INIT_ZMM avx512456
+cglobal blockcopy_ss_32x16, 4, 7, 4457
+458
+ add r1, r1459
+ add r3, r3460
+ lea r5, [3 * r1]461
+ lea r6, [3 * r3]462
+463
+ PROCESS_BLOCKCOPY_SS_W32_H8_avx512464
+ PROCESS_BLOCKCOPY_SS_W32_H8_LAST_avx512465
+ RET466
+467
+INIT_ZMM avx512468
+cglobal blockcopy_ss_32x24, 4, 7, 4469
+470
+ add r1, r1471
+ add r3, r3472
+ lea r5, [3 * r1]473
+ lea r6, [3 * r3]474
+475
+ PROCESS_BLOCKCOPY_SS_W32_H8_avx512476
+ PROCESS_BLOCKCOPY_SS_W32_H8_avx512477
+ PROCESS_BLOCKCOPY_SS_W32_H8_LAST_avx512478
+ RET479
+480
+INIT_ZMM avx512481
+cglobal blockcopy_ss_32x32, 4, 7, 4482
+483
+ add r1, r1484
+ add r3, r3485
+ lea r5, [3 * r1]486
+ lea r6, [3 * r3]487
+488
+ PROCESS_BLOCKCOPY_SS_W32_H8_avx512489
+ PROCESS_BLOCKCOPY_SS_W32_H8_avx512490
+ PROCESS_BLOCKCOPY_SS_W32_H8_avx512491
+ PROCESS_BLOCKCOPY_SS_W32_H8_LAST_avx512492
+ RET493
+494
+INIT_ZMM avx512495
+cglobal blockcopy_ss_32x48, 4, 7, 4496
+497
+ add r1, r1498
+ add r3, r3499
+ lea r5, [3 * r1]500
+ lea r6, [3 * r3]501
+502
+ PROCESS_BLOCKCOPY_SS_W32_H8_avx512503
+ PROCESS_BLOCKCOPY_SS_W32_H8_avx512504
+ PROCESS_BLOCKCOPY_SS_W32_H8_avx512505
+ PROCESS_BLOCKCOPY_SS_W32_H8_avx512506
+ PROCESS_BLOCKCOPY_SS_W32_H8_avx512507
+ PROCESS_BLOCKCOPY_SS_W32_H8_LAST_avx512508
+ RET509
+510
+INIT_ZMM avx512511
+cglobal blockcopy_ss_32x64, 4, 7, 4512
+513
+ add r1, r1514
+ add r3, r3515
+ lea r5, [3 * r1]516
+ lea r6, [3 * r3]517
+518
+ PROCESS_BLOCKCOPY_SS_W32_H8_avx512519
+ PROCESS_BLOCKCOPY_SS_W32_H8_avx512520
+ PROCESS_BLOCKCOPY_SS_W32_H8_avx512521
+ PROCESS_BLOCKCOPY_SS_W32_H8_avx512522
+ PROCESS_BLOCKCOPY_SS_W32_H8_avx512523
+ PROCESS_BLOCKCOPY_SS_W32_H8_avx512524
+ PROCESS_BLOCKCOPY_SS_W32_H8_avx512525
+ PROCESS_BLOCKCOPY_SS_W32_H8_LAST_avx512526
+ RET527
+528
;-----------------------------------------------------------------------------529
; void blockcopy_ss_%1x%2(int16_t* dst, intptr_t dstStride, const int16_t* src, intptr_t srcStride)530
;-----------------------------------------------------------------------------531
532
BLOCKCOPY_SS_W64_H4_avx 64, 48533
BLOCKCOPY_SS_W64_H4_avx 64, 64534
535
+%macro PROCESS_BLOCKCOPY_SS_W64_H8_avx512 0536
+ movu m0, [r2]537
+ movu m1, [r2 + mmsize]538
+ movu m2, [r2 + r3]539
+ movu m3, [r2 + r3 + mmsize]540
+541
+ movu [r0], m0542
+ movu [r0 + mmsize], m1543
+ movu [r0 + r1], m2544
+ movu [r0 + r1 + mmsize], m3545
+546
+ movu m0, [r2 + 2 * r3]547
+ movu m1, [r2 + 2 * r3 + mmsize]548
+ movu m2, [r2 + r6]549
+ movu m3, [r2 + r6 + mmsize]550
+ lea r2, [r2 + 4 * r3]551
+552
+ movu [r0 + 2 * r1], m0553
+ movu [r0 + 2 * r1 + mmsize], m1554
+ movu [r0 + r5], m2555
+ movu [r0 + r5 + mmsize], m3556
+ lea r0, [r0 + 4 * r1]557
+558
+ movu m0, [r2]559
+ movu m1, [r2 + mmsize]560
+ movu m2, [r2 + r3]561
+ movu m3, [r2 + r3 + mmsize]562
+563
+ movu [r0], m0564
+ movu [r0 + mmsize], m1565
+ movu [r0 + r1], m2566
+ movu [r0 + r1 + mmsize], m3567
+568
+ movu m0, [r2 + 2 * r3]569
+ movu m1, [r2 + 2 * r3 + mmsize]570
+ movu m2, [r2 + r6]571
+ movu m3, [r2 + r6 + mmsize]572
+ lea r2, [r2 + 4 * r3]573
+574
+ movu [r0 + 2 * r1], m0575
+ movu [r0 + 2 * r1 + mmsize], m1576
+ movu [r0 + r5], m2577
+ movu [r0 + r5 + mmsize], m3578
+ lea r0, [r0 + 4 * r1]579
+%endmacro580
+581
+%macro PROCESS_BLOCKCOPY_SS_W64_H8_LAST_avx512 0582
+ movu m0, [r2]583
+ movu m1, [r2 + mmsize]584
+ movu m2, [r2 + r3]585
+ movu m3, [r2 + r3 + mmsize]586
+587
+ movu [r0], m0588
+ movu [r0 + mmsize], m1589
+ movu [r0 + r1], m2590
+ movu [r0 + r1 + mmsize], m3591
+592
+ movu m0, [r2 + 2 * r3]593
+ movu m1, [r2 + 2 * r3 + mmsize]594
+ movu m2, [r2 + r6]595
+ movu m3, [r2 + r6 + mmsize]596
+ lea r2, [r2 + 4 * r3]597
+598
+ movu [r0 + 2 * r1], m0599
+ movu [r0 + 2 * r1 + mmsize], m1600
+ movu [r0 + r5], m2601
+ movu [r0 + r5 + mmsize], m3602
+ lea r0, [r0 + 4 * r1]603
+604
+ movu m0, [r2]605
+ movu m1, [r2 + mmsize]606
+ movu m2, [r2 + r3]607
+ movu m3, [r2 + r3 + mmsize]608
+609
+ movu [r0], m0610
+ movu [r0 + mmsize], m1611
+ movu [r0 + r1], m2612
+ movu [r0 + r1 + mmsize], m3613
+614
+ movu m0, [r2 + 2 * r3]615
+ movu m1, [r2 + 2 * r3 + mmsize]616
+ movu m2, [r2 + r6]617
+ movu m3, [r2 + r6 + mmsize]618
+619
+ movu [r0 + 2 * r1], m0620
+ movu [r0 + 2 * r1 + mmsize], m1621
+ movu [r0 + r5], m2622
+ movu [r0 + r5 + mmsize], m3623
+%endmacro624
+625
+;-----------------------------------------------------------------------------626
+; void blockcopy_ss_%1x%2(int16_t* dst, intptr_t dstStride, const int16_t* src, intptr_t srcStride)627
+;-----------------------------------------------------------------------------628
+INIT_ZMM avx512629
+cglobal blockcopy_ss_64x16, 4, 7, 4630
+ add r1, r1631
+ add r3, r3632
+ lea r5, [3 * r1]633
+ lea r6, [3 * r3]634
+635
+ PROCESS_BLOCKCOPY_SS_W64_H8_avx512636
+ PROCESS_BLOCKCOPY_SS_W64_H8_LAST_avx512637
+ RET638
+639
+INIT_ZMM avx512640
+cglobal blockcopy_ss_64x32, 4, 7, 4641
+ add r1, r1642
+ add r3, r3643
+ lea r5, [3 * r1]644
+ lea r6, [3 * r3]645
+646
+ PROCESS_BLOCKCOPY_SS_W64_H8_avx512647
+ PROCESS_BLOCKCOPY_SS_W64_H8_avx512648
+ PROCESS_BLOCKCOPY_SS_W64_H8_avx512649
+ PROCESS_BLOCKCOPY_SS_W64_H8_LAST_avx512650
+ RET651
+652
+INIT_ZMM avx512653
+cglobal blockcopy_ss_64x48, 4, 7, 4654
+ add r1, r1655
+ add r3, r3656
+ lea r5, [3 * r1]657
+ lea r6, [3 * r3]658
+659
+ PROCESS_BLOCKCOPY_SS_W64_H8_avx512660
+ PROCESS_BLOCKCOPY_SS_W64_H8_avx512661
+ PROCESS_BLOCKCOPY_SS_W64_H8_avx512662
+ PROCESS_BLOCKCOPY_SS_W64_H8_avx512663
+ PROCESS_BLOCKCOPY_SS_W64_H8_avx512664
+ PROCESS_BLOCKCOPY_SS_W64_H8_LAST_avx512665
+ RET666
+667
+INIT_ZMM avx512668
+cglobal blockcopy_ss_64x64, 4, 7, 4669
+ add r1, r1670
+ add r3, r3671
+ lea r5, [3 * r1]672
+ lea r6, [3 * r3]673
+674
+ PROCESS_BLOCKCOPY_SS_W64_H8_avx512675
+ PROCESS_BLOCKCOPY_SS_W64_H8_avx512676
+ PROCESS_BLOCKCOPY_SS_W64_H8_avx512677
+ PROCESS_BLOCKCOPY_SS_W64_H8_avx512678
+ PROCESS_BLOCKCOPY_SS_W64_H8_avx512679
+ PROCESS_BLOCKCOPY_SS_W64_H8_avx512680
+ PROCESS_BLOCKCOPY_SS_W64_H8_avx512681
+ PROCESS_BLOCKCOPY_SS_W64_H8_LAST_avx512682
+ RET683
;--------------------------------------------------------------------------------------684
; void cpy2Dto1D_shr(int16_t* dst, const int16_t* src, intptr_t srcStride, int shift);685
;--------------------------------------------------------------------------------------686
687
jnz .loop688
RET689
690
+INIT_ZMM avx512691
+cglobal cpy2Dto1D_shr_16, 4, 5, 4692
+ shl r2d, 1693
+ movd xm0, r3d694
+ pcmpeqw ymm1, ymm1695
+ psllw ym1, ymm1, xm0696
+ psraw ym1, 1697
+ vinserti32x8 m1, ym1, 1698
+ lea r3, [r2 * 3]699
+ mov r4d, 2700
+701
+.loop:702
+ ; Row 0-1703
+ movu ym2, [r1]704
+ vinserti32x8 m2, [r1 + r2], 1705
+ psubw m2, m1706
+ psraw m2, xm0707
+ movu [r0], m2708
+709
+ ; Row 2-3710
+ movu ym2, [r1 + 2 * r2]711
+ vinserti32x8 m2, [r1 + r3], 1712
+ psubw m2, m1713
+ psraw m2, xm0714
+ movu [r0 + mmsize], m2715
+716
+ lea r1, [r1 + 4 * r2]717
+ ; Row 4-5718
+719
+ movu ym2, [r1]720
+ vinserti32x8 m2, [r1 + r2], 1721
+ psubw m2, m1722
+ psraw m2, xm0723
+ movu [r0 + 2 * mmsize], m2724
+725
+ ; Row 6-7726
+ movu ym2, [r1 + 2 * r2]727
+ vinserti32x8 m2, [r1 + r3], 1728
+ psubw m2, m1729
+ psraw m2, xm0730
+ movu [r0 + 3 * mmsize], m2731
+732
+ add r0, 4 * mmsize733
+ lea r1, [r1 + 4 * r2]734
+ dec r4d735
+ jnz .loop736
+ RET737
738
;--------------------------------------------------------------------------------------739
; void cpy2Dto1D_shr(int16_t* dst, const int16_t* src, intptr_t srcStride, int shift);740
741
jnz .loop742
RET743
744
+INIT_ZMM avx512745
+cglobal cpy2Dto1D_shr_32, 4, 5, 4746
+ shl r2d, 1747
+ movd xm0, r3d748
+ pcmpeqw ymm1, ymm1749
+ psllw ym1, ymm1, xm0750
+ psraw ym1, 1751
+ vinserti32x8 m1, ym1, 1752
+ lea r3, [r2 * 3]753
+ mov r4d, 8754
+755
+.loop:756
+ ; Row 0757
+ movu m2, [r1]758
+ psubw m2, m1759
+ psraw m2, xm0760
+ movu [r0], m2761
+762
+ ; Row 1763
+ movu m2, [r1 + r2]764
+ psubw m2, m1765
+ psraw m2, xm0766
+ movu [r0 + mmsize], m2767
+768
+ ; Row 2769
+ movu m2, [r1 + 2 * r2]770
+ psubw m2, m1771
+ psraw m2, xm0772
+ movu [r0 + 2 * mmsize], m2773
+774
+ ; Row 3775
+ movu m2, [r1 + r3]776
+ psubw m2, m1777
+ psraw m2, xm0778
+ movu [r0 + 3 * mmsize], m2779
+780
+ add r0, 4 * mmsize781
+ lea r1, [r1 + 4 * r2]782
+ dec r4d783
+ jnz .loop784
+ RET785
+786
;--------------------------------------------------------------------------------------787
; void cpy1Dto2D_shl(int16_t* dst, const int16_t* src, intptr_t dstStride, int shift)788
;--------------------------------------------------------------------------------------789
790
jnz .loop791
RET792
793
+;--------------------------------------------------------------------------------------794
+; cpy_1Dto2D_shl avx512 code start795
+;--------------------------------------------------------------------------------------796
+%macro PROCESS_CPY1Dto2D_SHL_32x8_AVX512 0797
+ movu m1, [r1 + 0 * mmsize]798
+ movu m2, [r1 + 1 * mmsize]799
+ movu m3, [r1 + 2 * mmsize]800
+ movu m4, [r1 + 3 * mmsize]801
+ psllw m1, xm0802
+ psllw m2, xm0803
+ psllw m3, xm0804
+ psllw m4, xm0805
+ movu [r0], m1806
+ movu [r0 + r2], m2807
+ movu [r0 + 2 * r2], m3808
+ movu [r0 + r3], m4809
+810
+ add r1, 4 * mmsize811
+ lea r0, [r0 + r2 * 4]812
+813
+ movu m1, [r1 + 0 * mmsize]814
+ movu m2, [r1 + 1 * mmsize]815
+ movu m3, [r1 + 2 * mmsize]816
+ movu m4, [r1 + 3 * mmsize]817
+ psllw m1, xm0818
+ psllw m2, xm0819
+ psllw m3, xm0820
+ psllw m4, xm0821
+ movu [r0], m1822
+ movu [r0 + r2], m2823
+ movu [r0 + 2 * r2], m3824
+ movu [r0 + r3], m4825
+%endmacro826
+;--------------------------------------------------------------------------------------827
+; void cpy1Dto2D_shl(int16_t* dst, const int16_t* src, intptr_t dstStride, int shift)828
+;--------------------------------------------------------------------------------------829
+INIT_ZMM avx512830
+cglobal cpy1Dto2D_shl_32, 4, 4, 5831
+ add r2d, r2d832
+ movd xm0, r3d833
+ lea r3, [3 * r2]834
+%rep 3835
+ PROCESS_CPY1Dto2D_SHL_32x8_AVX512836
+ add r1, 4 * mmsize837
+ lea r0, [r0 + r2 * 4]838
+%endrep839
+ PROCESS_CPY1Dto2D_SHL_32x8_AVX512840
+ RET841
842
+%macro PROCESS_CPY1Dto2D_SHL_ALIGNED_32x8_AVX512 0843
+ mova m1, [r1 + 0 * mmsize]844
+ mova m2, [r1 + 1 * mmsize]845
+ mova m3, [r1 + 2 * mmsize]846
+ mova m4, [r1 + 3 * mmsize]847
+ psllw m1, xm0848
+ psllw m2, xm0849
+ psllw m3, xm0850
+ psllw m4, xm0851
+ mova [r0], m1852
+ mova [r0 + r2], m2853
+ mova [r0 + 2 * r2], m3854
+ mova [r0 + r3], m4855
+856
+ add r1, 4 * mmsize857
+ lea r0, [r0 + r2 * 4]858
+859
+ mova m1, [r1 + 0 * mmsize]860
+ mova m2, [r1 + 1 * mmsize]861
+ mova m3, [r1 + 2 * mmsize]862
+ mova m4, [r1 + 3 * mmsize]863
+ psllw m1, xm0864
+ psllw m2, xm0865
+ psllw m3, xm0866
+ psllw m4, xm0867
+ mova [r0], m1868
+ mova [r0 + r2], m2869
+ mova [r0 + 2 * r2], m3870
+ mova [r0 + r3], m4871
+%endmacro872
+;--------------------------------------------------------------------------------------873
+; void cpy1Dto2D_shl(int16_t* dst, const int16_t* src, intptr_t dstStride, int shift)874
+;--------------------------------------------------------------------------------------875
+INIT_ZMM avx512876
+cglobal cpy1Dto2D_shl_aligned_32, 4, 4, 5877
+ add r2d, r2d878
+ movd xm0, r3d879
+ lea r3, [3 * r2]880
+%rep 3881
+ PROCESS_CPY1Dto2D_SHL_ALIGNED_32x8_AVX512882
+ add r1, 4 * mmsize883
+ lea r0, [r0 + r2 * 4]884
+%endrep885
+ PROCESS_CPY1Dto2D_SHL_ALIGNED_32x8_AVX512886
+ RET887
+;--------------------------------------------------------------------------------------888
+; copy_cnt avx512 code end889
+;--------------------------------------------------------------------------------------890
;--------------------------------------------------------------------------------------891
; uint32_t copy_cnt(int16_t* dst, const int16_t* src, intptr_t srcStride);892
;--------------------------------------------------------------------------------------893
894
movd eax, xm4895
RET896
897
+;--------------------------------------------------------------------------------------898
+; copy_cnt avx512 code start899
+;--------------------------------------------------------------------------------------900
+%macro PROCESS_COPY_CNT_32x4_AVX512 0901
+ movu m0, [r1]902
+ movu m1, [r1 + r2]903
+ movu [r0], m0904
+ movu [r0 + mmsize], m1905
+ packsswb m0, m1906
+ pminub m0, m3907
+908
+ movu m1, [r1 + 2 * r2]909
+ movu m2, [r1 + r3]910
+ movu [r0 + 2 * mmsize], m1911
+ movu [r0 + 3 * mmsize], m2912
+ packsswb m1, m2913
+ pminub m1, m3914
+915
+ paddb m0, m1916
+ paddb m4, m0917
+%endmacro918
+919
+%macro PROCESS_COPY_CNT_16x4_AVX512 0920
+ movu ym0, [r1]921
+ vinserti32x8 m0, [r1 + r2], 1922
+ movu ym1, [r1 + 2 * r2]923
+ vinserti32x8 m1, [r1 + r3], 1924
+ movu [r0], m0925
+ movu [r0 + mmsize], m1926
+ packsswb m0, m1927
+ pminub m0, m3928
+ paddb m4, m0929
+%endmacro930
+931
+%macro PROCESS_COPY_CNT_END_AVX512 0932
+ pxor m0, m0933
+ vextracti32x8 ym1, m4, 1934
+ paddb ym4, ym1935
+ vextracti32x4 xm1, ym4, 1936
+ paddb xm4, xm1937
+ psadbw xm4, xm0938
+ movhlps xm1, xm4939
+ paddd xm4, xm1940
+ movd eax, xm4941
+%endmacro942
+943
+;--------------------------------------------------------------------------------------944
+; uint32_t copy_cnt(int32_t* dst, const int16_t* src, intptr_t stride);945
+;--------------------------------------------------------------------------------------946
+INIT_ZMM avx512947
+cglobal copy_cnt_32, 3, 4, 5948
+ add r2d, r2d949
+ lea r3, [3 * r2]950
+951
+ vbroadcasti32x8 m3, [pb_1]952
+ pxor m4, m4953
+954
+%rep 7955
+ PROCESS_COPY_CNT_32x4_AVX512956
+ add r0, 4 * mmsize957
+ lea r1, [r1 + 4 * r2]958
+%endrep959
+ PROCESS_COPY_CNT_32x4_AVX512960
+ PROCESS_COPY_CNT_END_AVX512961
+ RET962
+963
+INIT_ZMM avx512964
+cglobal copy_cnt_16, 3, 4, 5965
+ add r2d, r2d966
+ lea r3, [3 * r2]967
+968
+ vbroadcasti32x8 m3, [pb_1]969
+ pxor m4, m4970
971
+%rep 3972
+ PROCESS_COPY_CNT_16x4_AVX512973
+ add r0, 2 * mmsize974
+ lea r1, [r1 + 4 * r2]975
+%endrep976
+ PROCESS_COPY_CNT_16x4_AVX512977
+ PROCESS_COPY_CNT_END_AVX512978
+ RET979
+;--------------------------------------------------------------------------------------980
+; copy_cnt avx512 code end981
+;--------------------------------------------------------------------------------------982
;--------------------------------------------------------------------------------------983
; void cpy2Dto1D_shl(int16_t* dst, const int16_t* src, intptr_t srcStride, int shift);984
;--------------------------------------------------------------------------------------985
986
RET987
988
;--------------------------------------------------------------------------------------989
+; cpy2Dto1D_shl avx512 code start990
+;--------------------------------------------------------------------------------------991
+%macro PROCESS_CPY2Dto1D_SHL_16x8_AVX512 0992
+ movu m1, [r1]993
+ vinserti32x8 m1, [r1 + r2], 1994
+ movu m2, [r1 + 2 * r2]995
+ vinserti32x8 m2, [r1 + r3], 1996
+997
+ psllw m1, xm0998
+ psllw m2, xm0999
+ movu [r0], m11000
+ movu [r0 + mmsize], m21001
+1002
+ add r0, 2 * mmsize1003
+ lea r1, [r1 + r2 * 4]1004
+1005
+ movu m1, [r1]1006
+ vinserti32x8 m1, [r1 + r2], 11007
+ movu m2, [r1 + 2 * r2]1008
+ vinserti32x8 m2, [r1 + r3], 11009
+1010
+ psllw m1, xm01011
+ psllw m2, xm01012
+ movu [r0], m11013
+ movu [r0 + mmsize], m21014
+%endmacro1015
+1016
+%macro PROCESS_CPY2Dto1D_SHL_32x8_AVX512 01017
+ movu m1, [r1]1018
+ movu m2, [r1 + r2]1019
+ movu m3, [r1 + 2 * r2]1020
+ movu m4, [r1 + r3]1021
+1022
+ psllw m1, xm01023
+ psllw m2, xm01024
+ psllw m3, xm01025
+ psllw m4, xm01026
+ movu [r0], m11027
+ movu [r0 + mmsize], m21028
+ movu [r0 + 2 * mmsize], m31029
+ movu [r0 + 3 * mmsize], m41030
+1031
+ add r0, 4 * mmsize1032
+ lea r1, [r1 + r2 * 4]1033
+1034
+ movu m1, [r1]1035
+ movu m2, [r1 + r2]1036
+ movu m3, [r1 + 2 * r2]1037
+ movu m4, [r1 + r3]1038
+1039
+ psllw m1, xm01040
+ psllw m2, xm01041
+ psllw m3, xm01042
+ psllw m4, xm01043
+ movu [r0], m11044
+ movu [r0 + mmsize], m21045
+ movu [r0 + 2 * mmsize], m31046
+ movu [r0 + 3 * mmsize], m41047
+%endmacro1048
+1049
+;--------------------------------------------------------------------------------------1050
+; void cpy2Dto1D_shl(int16_t* dst, const int16_t* src, intptr_t srcStride, int shift);1051
+;--------------------------------------------------------------------------------------1052
+INIT_ZMM avx5121053
+cglobal cpy2Dto1D_shl_32, 4, 4, 51054
+ add r2d, r2d1055
+ movd xm0, r3d1056
+ lea r3, [3 * r2]1057
+1058
+ PROCESS_CPY2Dto1D_SHL_32x8_AVX5121059
+ add r0, 4 * mmsize1060
+ lea r1, [r1 + r2 * 4]1061
+ PROCESS_CPY2Dto1D_SHL_32x8_AVX5121062
+ add r0, 4 * mmsize1063
+ lea r1, [r1 + r2 * 4]1064
+ PROCESS_CPY2Dto1D_SHL_32x8_AVX5121065
+ add r0, 4 * mmsize1066
+ lea r1, [r1 + r2 * 4]1067
+ PROCESS_CPY2Dto1D_SHL_32x8_AVX5121068
+ RET1069
+1070
+INIT_ZMM avx5121071
+cglobal cpy2Dto1D_shl_16, 4, 4, 31072
+ add r2d, r2d1073
+ movd xm0, r3d1074
+ lea r3, [3 * r2]1075
+1076
+ PROCESS_CPY2Dto1D_SHL_16x8_AVX5121077
+ add r0, 2 * mmsize1078
+ lea r1, [r1 + r2 * 4]1079
+ PROCESS_CPY2Dto1D_SHL_16x8_AVX5121080
+ RET1081
+;--------------------------------------------------------------------------------------1082
+; cpy2Dto1D_shl avx512 code end1083
+;--------------------------------------------------------------------------------------1084
+;--------------------------------------------------------------------------------------1085
; void cpy1Dto2D_shr(int16_t* dst, const int16_t* src, intptr_t dstStride, int shift)1086
;--------------------------------------------------------------------------------------1087
INIT_XMM sse21088
1089
jnz .loop1090
RET1091
1092
+INIT_ZMM avx5121093
+cglobal cpy1Dto2D_shr_16, 3, 5, 41094
+ shl r2d, 11095
+ movd xm0, r3m1096
+ pcmpeqw xmm1, xmm11097
+ psllw xm1, xmm1, xm01098
+ psraw xm1, 11099
+ vpbroadcastw m1, xm11100
+ mov r3d, 41101
+ lea r4, [r2 * 3]1102
+1103
+.loop:1104
+ ; Row 0-11105
+ movu m2, [r1]1106
+ psubw m2, m11107
+ psraw m2, xm01108
+ movu [r0], ym21109
+ vextracti32x8 [r0 + r2], m2, 11110
+1111
+ ; Row 2-31112
+ movu m2, [r1 + mmsize]1113
+ psubw m2, m11114
+ psraw m2, xm01115
+ movu [r0 + r2 * 2], ym21116
+ vextracti32x8 [r0 + r4], m2, 11117
+1118
+ add r1, 2 * mmsize1119
+ lea r0, [r0 + r2 * 4]1120
+ dec r3d1121
+ jnz .loop1122
+ RET1123
1124
;--------------------------------------------------------------------------------------1125
; void cpy1Dto2D_shr(int16_t* dst, const int16_t* src, intptr_t dstStride, int shift)1126
1127
dec r3d1128
jnz .loop1129
RET1130
+1131
+INIT_ZMM avx5121132
+cglobal cpy1Dto2D_shr_32, 3, 4, 61133
+ shl r2d, 11134
+ movd xm0, r3m1135
+ pcmpeqw xmm1, xmm11136
+ psllw xm1, xmm1, xm01137
+ psraw xm1, 11138
+ vpbroadcastw m1, xm11139
+ mov r3d, 161140
+1141
+.loop:1142
+ ; Row 0-11143
+ movu m2, [r1]1144
+ movu m3, [r1 + mmsize]1145
+ psubw m2, m11146
+ psubw m3, m11147
+ psraw m2, xm01148
+ psraw m3, xm01149
+ movu [r0], m21150
+ movu [r0 + r2], m31151
+1152
+ add r1, 2 * mmsize1153
+ lea r0, [r0 + r2 * 2]1154
+ dec r3d1155
+ jnz .loop1156
+ RET1157
x265_2.7.tar.gz/source/common/x86/blockcopy8.h -> x265_2.9.tar.gz/source/common/x86/blockcopy8.h
Changed
51
1
2
FUNCDEF_TU_S(void, cpy2Dto1D_shl, sse2, int16_t* dst, const int16_t* src, intptr_t srcStride, int shift);3
FUNCDEF_TU_S(void, cpy2Dto1D_shl, sse4, int16_t* dst, const int16_t* src, intptr_t srcStride, int shift);4
FUNCDEF_TU_S(void, cpy2Dto1D_shl, avx2, int16_t* dst, const int16_t* src, intptr_t srcStride, int shift);5
+FUNCDEF_TU_S(void, cpy2Dto1D_shl, avx512, int16_t* dst, const int16_t* src, intptr_t srcStride, int shift);6
7
FUNCDEF_TU_S(void, cpy2Dto1D_shr, sse2, int16_t* dst, const int16_t* src, intptr_t srcStride, int shift);8
FUNCDEF_TU_S(void, cpy2Dto1D_shr, sse4, int16_t* dst, const int16_t* src, intptr_t srcStride, int shift);9
FUNCDEF_TU_S(void, cpy2Dto1D_shr, avx2, int16_t* dst, const int16_t* src, intptr_t srcStride, int shift);10
+FUNCDEF_TU_S(void, cpy2Dto1D_shr, avx512, int16_t* dst, const int16_t* src, intptr_t srcStride, int shift);11
12
FUNCDEF_TU_S(void, cpy1Dto2D_shl, sse2, int16_t* dst, const int16_t* src, intptr_t srcStride, int shift);13
FUNCDEF_TU_S(void, cpy1Dto2D_shl, sse4, int16_t* dst, const int16_t* src, intptr_t srcStride, int shift);14
FUNCDEF_TU_S(void, cpy1Dto2D_shl, avx2, int16_t* dst, const int16_t* src, intptr_t srcStride, int shift);15
-16
+FUNCDEF_TU_S(void, cpy1Dto2D_shl, avx512, int16_t* dst, const int16_t* src, intptr_t srcStride, int shift);17
+FUNCDEF_TU_S(void, cpy1Dto2D_shl_aligned, avx512, int16_t* dst, const int16_t* src, intptr_t srcStride, int shift);18
FUNCDEF_TU_S(void, cpy1Dto2D_shr, sse2, int16_t* dst, const int16_t* src, intptr_t srcStride, int shift);19
FUNCDEF_TU_S(void, cpy1Dto2D_shr, sse4, int16_t* dst, const int16_t* src, intptr_t srcStride, int shift);20
FUNCDEF_TU_S(void, cpy1Dto2D_shr, avx2, int16_t* dst, const int16_t* src, intptr_t srcStride, int shift);21
+FUNCDEF_TU_S(void, cpy1Dto2D_shr, avx512, int16_t* dst, const int16_t* src, intptr_t srcStride, int shift);22
23
FUNCDEF_TU_S(uint32_t, copy_cnt, sse2, int16_t* dst, const int16_t* src, intptr_t srcStride);24
FUNCDEF_TU_S(uint32_t, copy_cnt, sse4, int16_t* dst, const int16_t* src, intptr_t srcStride);25
FUNCDEF_TU_S(uint32_t, copy_cnt, avx2, int16_t* dst, const int16_t* src, intptr_t srcStride);26
+FUNCDEF_TU_S(uint32_t, copy_cnt, avx512, int16_t* dst, const int16_t* src, intptr_t srcStride);27
28
FUNCDEF_TU(void, blockfill_s, sse2, int16_t* dst, intptr_t dstride, int16_t val);29
FUNCDEF_TU(void, blockfill_s, avx2, int16_t* dst, intptr_t dstride, int16_t val);30
+FUNCDEF_TU(void, blockfill_s, avx512, int16_t* dst, intptr_t dstride, int16_t val);31
+FUNCDEF_TU(void, blockfill_s_aligned, avx512, int16_t* dst, intptr_t dstride, int16_t val);32
33
FUNCDEF_CHROMA_PU(void, blockcopy_ss, sse2, int16_t* dst, intptr_t dstStride, const int16_t* src, intptr_t srcStride);34
FUNCDEF_CHROMA_PU(void, blockcopy_ss, avx, int16_t* dst, intptr_t dstStride, const int16_t* src, intptr_t srcStride);35
+FUNCDEF_CHROMA_PU(void, blockcopy_ss, avx512, int16_t* dst, intptr_t dstStride, const int16_t* src, intptr_t srcStride);36
37
FUNCDEF_CHROMA_PU(void, blockcopy_pp, sse2, pixel* dst, intptr_t dstStride, const pixel* src, intptr_t srcStride);38
FUNCDEF_CHROMA_PU(void, blockcopy_pp, avx, pixel* dst, intptr_t dstStride, const pixel* src, intptr_t srcStride);39
+FUNCDEF_CHROMA_PU(void, blockcopy_pp, avx512, pixel* dst, intptr_t dstStride, const pixel* src, intptr_t srcStride);40
41
FUNCDEF_PU(void, blockcopy_sp, sse2, pixel* dst, intptr_t dstStride, const int16_t* src, intptr_t srcStride);42
FUNCDEF_PU(void, blockcopy_sp, sse4, pixel* dst, intptr_t dstStride, const int16_t* src, intptr_t srcStride);43
FUNCDEF_PU(void, blockcopy_sp, avx2, pixel* dst, intptr_t dstStride, const int16_t* src, intptr_t srcStride);44
+FUNCDEF_PU(void, blockcopy_sp, avx512, pixel* dst, intptr_t dstStride, const int16_t* src, intptr_t srcStride);45
FUNCDEF_PU(void, blockcopy_ps, sse2, int16_t* dst, intptr_t dstStride, const pixel* src, intptr_t srcStride);46
FUNCDEF_PU(void, blockcopy_ps, sse4, int16_t* dst, intptr_t dstStride, const pixel* src, intptr_t srcStride);47
FUNCDEF_PU(void, blockcopy_ps, avx2, int16_t* dst, intptr_t dstStride, const pixel* src, intptr_t srcStride);48
+FUNCDEF_PU(void, blockcopy_ps, avx512, int16_t* dst, intptr_t dstStride, const pixel* src, intptr_t srcStride);49
50
#endif // ifndef X265_I386_PIXEL_H51
x265_2.7.tar.gz/source/common/x86/const-a.asm -> x265_2.9.tar.gz/source/common/x86/const-a.asm
Changed
10
1
2
3
%include "x86inc.asm"4
5
-SECTION_RODATA 326
+SECTION_RODATA 647
8
;; 8-bit constants9
10
x265_2.7.tar.gz/source/common/x86/cpu-a.asm -> x265_2.9.tar.gz/source/common/x86/cpu-a.asm
Changed
46
1
2
RET3
4
;-----------------------------------------------------------------------------5
-; void cpu_xgetbv( int op, int *eax, int *edx )6
+; uint64_t cpu_xgetbv( int xcr )7
;-----------------------------------------------------------------------------8
-cglobal cpu_xgetbv, 3,79
- push r210
- push r111
- mov ecx, r0d12
+cglobal cpu_xgetbv13
+ movifnidn ecx, r0m14
xgetbv15
- pop r416
- mov [r4], eax17
- pop r418
- mov [r4], edx19
- RET20
+%if ARCH_X86_6421
+ shl rdx, 3222
+ or rax, rdx23
+%endif24
+ ret25
26
%if ARCH_X86_6427
28
29
%if WIN6430
sub rsp, 32 ; shadow space31
%endif32
- and rsp, ~3133
+ and rsp, ~(STACK_ALIGNMENT - 1)34
mov rax, r035
mov r0, r136
mov r1, r237
38
push ebp39
mov ebp, esp40
sub esp, 1241
- and esp, ~3142
+ and esp, ~(STACK_ALIGNMENT - 1)43
mov ecx, [ebp+8]44
mov edx, [ebp+12]45
mov [esp], edx46
x265_2.7.tar.gz/source/common/x86/dct8.asm -> x265_2.9.tar.gz/source/common/x86/dct8.asm
Changed
4080
1
2
3
%include "x86inc.asm"4
%include "x86util.asm"5
-SECTION_RODATA 326
+SECTION_RODATA 647
+8
+tab_dct32: dw 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 649
+ dw 90, 90, 88, 85, 82, 78, 73, 67, 61, 54, 46, 38, 31, 22, 13, 4, -4, -13, -22, -31, -38, -46, -54, -61, -67, -73, -78, -82, -85, -88, -90, -9010
+ dw 90, 87, 80, 70, 57, 43, 25, 9, -9, -25, -43, -57, -70, -80, -87, -90, -90, -87, -80, -70, -57, -43, -25, -9, 9, 25, 43, 57, 70, 80, 87, 9011
+ dw 90, 82, 67, 46, 22, -4, -31, -54, -73, -85, -90, -88, -78, -61, -38, -13, 13, 38, 61, 78, 88, 90, 85, 73, 54, 31, 4, -22, -46, -67, -82, -9012
+ dw 89, 75, 50, 18, -18, -50, -75, -89, -89, -75, -50, -18, 18, 50, 75, 89, 89, 75, 50, 18, -18, -50, -75, -89, -89, -75, -50, -18, 18, 50, 75, 8913
+ dw 88, 67, 31, -13, -54, -82, -90, -78, -46, -4, 38, 73, 90, 85, 61, 22, -22, -61, -85, -90, -73, -38, 4, 46, 78, 90, 82, 54, 13, -31, -67, -8814
+ dw 87, 57, 9, -43, -80, -90, -70, -25, 25, 70, 90, 80, 43, -9, -57, -87, -87, -57, -9, 43, 80, 90, 70, 25, -25, -70, -90, -80, -43, 9, 57, 8715
+ dw 85, 46, -13, -67, -90, -73, -22, 38, 82, 88, 54, -4, -61, -90, -78, -31, 31, 78, 90, 61, 4, -54, -88, -82, -38, 22, 73, 90, 67, 13, -46, -8516
+ dw 83, 36, -36, -83, -83, -36, 36, 83, 83, 36, -36, -83, -83, -36, 36, 83, 83, 36, -36, -83, -83, -36, 36, 83, 83, 36, -36, -83, -83, -36, 36, 8317
+ dw 82, 22, -54, -90, -61, 13, 78, 85, 31, -46, -90, -67, 4, 73, 88, 38, -38, -88, -73, -4, 67, 90, 46, -31, -85, -78, -13, 61, 90, 54, -22, -8218
+ dw 80, 9, -70, -87, -25, 57, 90, 43, -43, -90, -57, 25, 87, 70, -9, -80, -80, -9, 70, 87, 25, -57, -90, -43, 43, 90, 57, -25, -87, -70, 9, 8019
+ dw 78, -4, -82, -73, 13, 85, 67, -22, -88, -61, 31, 90, 54, -38, -90, -46, 46, 90, 38, -54, -90, -31, 61, 88, 22, -67, -85, -13, 73, 82, 4, -7820
+ dw 75, -18, -89, -50, 50, 89, 18, -75, -75, 18, 89, 50, -50, -89, -18, 75, 75, -18, -89, -50, 50, 89, 18, -75, -75, 18, 89, 50, -50, -89, -18, 7521
+ dw 73, -31, -90, -22, 78, 67, -38, -90, -13, 82, 61, -46, -88, -4, 85, 54, -54, -85, 4, 88, 46, -61, -82, 13, 90, 38, -67, -78, 22, 90, 31, -7322
+ dw 70, -43, -87, 9, 90, 25, -80, -57, 57, 80, -25, -90, -9, 87, 43, -70, -70, 43, 87, -9, -90, -25, 80, 57, -57, -80, 25, 90, 9, -87, -43, 7023
+ dw 67, -54, -78, 38, 85, -22, -90, 4, 90, 13, -88, -31, 82, 46, -73, -61, 61, 73, -46, -82, 31, 88, -13, -90, -4, 90, 22, -85, -38, 78, 54, -6724
+ dw 64, -64, -64, 64, 64, -64, -64, 64, 64, -64, -64, 64, 64, -64, -64, 64, 64, -64, -64, 64, 64, -64, -64, 64, 64, -64, -64, 64, 64, -64, -64, 6425
+ dw 61, -73, -46, 82, 31, -88, -13, 90, -4, -90, 22, 85, -38, -78, 54, 67, -67, -54, 78, 38, -85, -22, 90, 4, -90, 13, 88, -31, -82, 46, 73, -6126
+ dw 57, -80, -25, 90, -9, -87, 43, 70, -70, -43, 87, 9, -90, 25, 80, -57, -57, 80, 25, -90, 9, 87, -43, -70, 70, 43, -87, -9, 90, -25, -80, 5727
+ dw 54, -85, -4, 88, -46, -61, 82, 13, -90, 38, 67, -78, -22, 90, -31, -73, 73, 31, -90, 22, 78, -67, -38, 90, -13, -82, 61, 46, -88, 4, 85, -5428
+ dw 50, -89, 18, 75, -75, -18, 89, -50, -50, 89, -18, -75, 75, 18, -89, 50, 50, -89, 18, 75, -75, -18, 89, -50, -50, 89, -18, -75, 75, 18, -89, 5029
+ dw 46, -90, 38, 54, -90, 31, 61, -88, 22, 67, -85, 13, 73, -82, 4, 78, -78, -4, 82, -73, -13, 85, -67, -22, 88, -61, -31, 90, -54, -38, 90, -4630
+ dw 43, -90, 57, 25, -87, 70, 9, -80, 80, -9, -70, 87, -25, -57, 90, -43, -43, 90, -57, -25, 87, -70, -9, 80, -80, 9, 70, -87, 25, 57, -90, 4331
+ dw 38, -88, 73, -4, -67, 90, -46, -31, 85, -78, 13, 61, -90, 54, 22, -82, 82, -22, -54, 90, -61, -13, 78, -85, 31, 46, -90, 67, 4, -73, 88, -3832
+ dw 36, -83, 83, -36, -36, 83, -83, 36, 36, -83, 83, -36, -36, 83, -83, 36, 36, -83, 83, -36, -36, 83, -83, 36, 36, -83, 83, -36, -36, 83, -83, 3633
+ dw 31, -78, 90, -61, 4, 54, -88, 82, -38, -22, 73, -90, 67, -13, -46, 85, -85, 46, 13, -67, 90, -73, 22, 38, -82, 88, -54, -4, 61, -90, 78, -3134
+ dw 25, -70, 90, -80, 43, 9, -57, 87, -87, 57, -9, -43, 80, -90, 70, -25, -25, 70, -90, 80, -43, -9, 57, -87, 87, -57, 9, 43, -80, 90, -70, 2535
+ dw 22, -61, 85, -90, 73, -38, -4, 46, -78, 90, -82, 54, -13, -31, 67, -88, 88, -67, 31, 13, -54, 82, -90, 78, -46, 4, 38, -73, 90, -85, 61, -2236
+ dw 18, -50, 75, -89, 89, -75, 50, -18, -18, 50, -75, 89, -89, 75, -50, 18, 18, -50, 75, -89, 89, -75, 50, -18, -18, 50, -75, 89, -89, 75, -50, 1837
+ dw 13, -38, 61, -78, 88, -90, 85, -73, 54, -31, 4, 22, -46, 67, -82, 90, -90, 82, -67, 46, -22, -4, 31, -54, 73, -85, 90, -88, 78, -61, 38, -1338
+ dw 9, -25, 43, -57, 70, -80, 87, -90, 90, -87, 80, -70, 57, -43, 25, -9, -9, 25, -43, 57, -70, 80, -87, 90, -90, 87, -80, 70, -57, 43, -25, 939
+ dw 4, -13, 22, -31, 38, -46, 54, -61, 67, -73, 78, -82, 85, -88, 90, -90, 90, -90, 88, -85, 82, -78, 73, -67, 61, -54, 46, -38, 31, -22, 13, -440
+tab_dct16: dw 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 6441
+ dw 90, 87, 80, 70, 57, 43, 25, 9, -9, -25, -43, -57, -70, -80, -87, -9042
+ dw 89, 75, 50, 18, -18, -50, -75, -89, -89, -75, -50, -18, 18, 50, 75, 8943
+ dw 87, 57, 9, -43, -80, -90, -70, -25, 25, 70, 90, 80, 43, -9, -57, -8744
+ dw 83, 36, -36, -83, -83, -36, 36, 83, 83, 36, -36, -83, -83, -36, 36, 8345
+ dw 80, 9, -70, -87, -25, 57, 90, 43, -43, -90, -57, 25, 87, 70, -9, -8046
+ dw 75, -18, -89, -50, 50, 89, 18, -75, -75, 18, 89, 50, -50, -89, -18, 7547
+ dw 70, -43, -87, 9, 90, 25, -80, -57, 57, 80, -25, -90, -9, 87, 43, -7048
+ dw 64, -64, -64, 64, 64, -64, -64, 64, 64, -64, -64, 64, 64, -64, -64, 6449
+ dw 57, -80, -25, 90, -9, -87, 43, 70, -70, -43, 87, 9, -90, 25, 80, -5750
+ dw 50, -89, 18, 75, -75, -18, 89, -50, -50, 89, -18, -75, 75, 18, -89, 5051
+ dw 43, -90, 57, 25, -87, 70, 9, -80, 80, -9, -70, 87, -25, -57, 90, -4352
+ dw 36, -83, 83, -36, -36, 83, -83, 36, 36, -83, 83, -36, -36, 83, -83, 3653
+ dw 25, -70, 90, -80, 43, 9, -57, 87, -87, 57, -9, -43, 80, -90, 70, -2554
+ dw 18, -50, 75, -89, 89, -75, 50, -18, -18, 50, -75, 89, -89, 75, -50, 1855
+ dw 9, -25, 43, -57, 70, -80, 87, -90, 90, -87, 80, -70, 57, -43, 25, -956
+57
+dct16_shuf_AVX512: dq 0, 1, 8, 9, 4, 5, 12, 1358
+dct16_shuf1_AVX512: dq 2, 3, 10, 11, 6, 7, 14, 1559
+dct16_shuf3_AVX512: dq 0, 1, 4, 5, 8, 9, 12, 1360
+dct16_shuf4_AVX512: dq 2, 3, 6, 7, 10, 11, 14, 1561
+dct16_shuf2_AVX512: dd 0, 4, 8, 12, 2, 6, 10, 14, 16, 20, 24, 28, 18, 22, 26, 3062
+63
+dct8_shuf5_AVX512: dq 0, 2, 4, 6, 1, 3, 5, 764
+dct8_shuf6_AVX512: dq 0, 2, 4, 6, 1, 3, 5, 765
+dct8_shuf8_AVX512: dd 0, 2, 8, 10, 4, 6, 12, 14, 1, 3, 5, 7, 9, 11, 13, 1566
+dct8_shuf4_AVX512: times 2 dd 0, 2, 4, 6, 8, 10, 12, 14, 1, 3, 5, 7, 9, 11, 13, 1567
+dct16_shuf7_AVX512: dd 0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 3068
+dct16_shuf9_AVX512: dd 0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 1569
+70
+dct32_shuf_AVX512: dd 0, 1, 4, 5, 8, 9, 12, 13, 16, 17, 20 , 21, 24, 25, 28, 2971
+dct32_shuf4_AVX512: times 2 dd 0, 4, 8, 12, 0, 4, 8, 1272
+dct32_shuf5_AVX512: dd 0, 0, 0, 0, -1, -1, -1, -1, 0, 0, 0, 0, 0, 0, 0, 073
+dct32_shuf6_AVX512: dd 0, 0, 0, 0, 0, 0, 0, 0, -1, -1, -1, -1, 0, 0, 0, 074
+dct32_shuf7_AVX512: dd 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -1, -1, -1, -175
+dct32_shuf8_AVX512: dd -1, -1, -1, -1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 076
+dct16_shuf5_AVX512: dw 0, 1, 2, 3, 8, 9, 10, 11, 16, 17, 18, 19, 24, 25, 26, 27, 4, 5, 6, 7, 12, 13, 14, 15, 20, 21, 22, 23, 28, 29, 30, 3177
+dct16_shuf6_AVX512: dw 0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 3078
+dct16_shuf8_AVX512: dw 20, 0, 4, 2, 28, 8, 6, 10, 22, 16, 12, 18, 30, 24, 14, 2679
+80
+dct8_shuf7_AVX512: dw 0, 2, 16, 18, 8, 10, 24, 26, 4, 6, 20, 22, 12, 14, 28, 3081
+dct8_shuf9_AVX512: times 2 dw 0, 8, 16, 24, 4, 12, 20, 2882
+dct32_shuf1_AVX512: dw 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0, 31, 30, 29, 28, 27, 26, 25, 24, 23, 22, 21, 20, 19, 18, 17, 1683
+dct32_shuf2_AVX512: dw 0, 1, 2, 3, 4, 5, 6, 7, 16, 17, 18, 19, 20, 21, 22, 23, 15, 14, 13, 12, 11, 10, 9, 8, 31, 30, 29, 28, 27, 26, 25, 2484
+dct32_shuf3_AVX512: times 2 dw 0, 8, 16, 24, 2, 10, 18, 2685
+86
+dct8_shuf: times 2 db 6, 7, 4, 5, 2, 3, 0, 1, 14, 15, 12, 13, 10, 11, 8, 987
+dct8_shuf_AVX512: times 2 db 4, 5, 6, 7, 0, 1, 2, 3, 12, 13, 14, 15, 8, 9, 10, 1188
+89
tab_dct8: dw 64, 64, 64, 64, 64, 64, 64, 6490
dw 89, 75, 50, 18, -18, -50, -75, -8991
dw 83, 36, -36, -83, -83, -36, 36, 8392
93
dw 36, -83, 83, -36, -36, 83, -83, 3694
dw 18, -50, 75, -89, 89, -75, 50, -1895
96
-dct8_shuf: times 2 db 6, 7, 4, 5, 2, 3, 0, 1, 14, 15, 12, 13, 10, 11, 8, 997
+tab_dct8_avx512: dw 64, 64, 64, 64, 89, 75, 50, 1898
+ dw 83, 36, -36, -83, 75, -18, -89, -5099
+ dw 64, -64, -64, 64, 50, -89, 18, 75100
+ dw 36, -83, 83, -36, 18, -50, 75, -89101
102
tab_dct16_1: dw 64, 64, 64, 64, 64, 64, 64, 64103
dw 90, 87, 80, 70, 57, 43, 25, 9104
105
dw 18, -50, 75, -89, 89, -75, 50, -18106
dw 9, -25, 43, -57, 70, -80, 87, -90107
108
-109
tab_dct16_2: dw 64, 64, 64, 64, 64, 64, 64, 64110
dw -9, -25, -43, -57, -70, -80, -87, -90111
dw -89, -75, -50, -18, 18, 50, 75, 89112
113
times 4 dw 50, -89, 18, 75114
times 4 dw 18, -50, 75, -89115
116
+avx512_idct8_1: times 8 dw 64, 83, 64, 36117
+ times 8 dw 64, 36, -64, -83118
+ times 8 dw 64, -36, -64, 83119
+ times 8 dw 64, -83, 64, -36120
+121
+avx512_idct8_2: times 8 dw 89, 75, 50, 18122
+ times 8 dw 75, -18, -89, -50123
+ times 8 dw 50, -89, 18, 75124
+ times 8 dw 18, -50, 75, -89125
+126
+avx512_idct8_3: dw 64, 83, 64, 83, 64, 83, 64, 83, 64, 83, 64, 83, 64, 83, 64, 83, 64, -36, 64, -36, 64, -36, 64, -36, 64, -36, 64, -36, 64, -36, 64, -36127
+ dw 64, 36, 64, 36, 64, 36, 64, 36, 64, 36, 64, 36, 64, 36, 64, 36, -64, 83, -64, 83, -64, 83, -64, 83, -64, 83, -64, 83, -64, 83, -64, 83128
+ dw 64, 36, 64, 36, 64, 36, 64, 36, 64, 36, 64, 36, 64, 36, 64, 36, 64, -83, 64, -83, 64, -83, 64, -83, 64, -83, 64, -83, 64, -83, 64, -83129
+ dw -64, -83, -64, -83, -64, -83, -64, -83, -64, -83, -64, -83, -64, -83, -64, -83, 64, -36, 64, -36, 64, -36, 64, -36, 64, -36, 64, -36, 64, -36, 64, -36130
+ dw 89, 75, 89, 75, 89, 75, 89, 75, 89, 75, 89, 75, 89, 75, 89, 75, 50, -89, 50, -89, 50, -89, 50, -89, 50, -89, 50, -89, 50, -89, 50, -89131
+ dw 50, 18, 50, 18, 50, 18, 50, 18, 50, 18, 50, 18, 50, 18, 50, 18, 18, 75, 18, 75, 18, 75, 18, 75, 18, 75, 18, 75, 18, 75, 18, 75132
+ dw 75, -18, 75, -18, 75, -18, 75, -18, 75, -18, 75, -18, 75, -18, 75, -18, 18, -50, 18, -50, 18, -50, 18, -50, 18, -50, 18, -50, 18, -50, 18, -50133
+ dw -89, -50, -89, -50, -89, -50, -89, -50, -89, -50, -89, -50, -89, -50, -89, -50, 75, -89, 75, -89, 75, -89, 75, -89, 75, -89, 75, -89, 75, -89, 75, -89134
+135
idct8_shuf1: dd 0, 2, 4, 6, 1, 3, 5, 7136
137
const idct8_shuf2, times 2 db 0, 1, 2, 3, 8, 9, 10, 11, 4, 5, 6, 7, 12, 13, 14, 15138
139
idct8_shuf3: times 2 db 12, 13, 14, 15, 8, 9, 10, 11, 4, 5, 6, 7, 0, 1, 2, 3140
141
+142
+idct8_avx512_shuf3: times 4 db 12, 13, 14, 15, 8, 9, 10, 11, 4, 5, 6, 7, 0, 1, 2, 3143
+144
tab_idct16_1: dw 90, 87, 80, 70, 57, 43, 25, 9145
dw 87, 57, 9, -43, -80, -90, -70, -25146
dw 80, 9, -70, -87, -25, 57, 90, 43147
148
idct16_shuff: dd 0, 4, 2, 6, 1, 5, 3, 7149
150
idct16_shuff1: dd 2, 6, 0, 4, 3, 7, 1, 5151
+idct16_shuff2: dw 0, 16, 2, 18, 4, 20, 6, 22, 8, 24, 10, 26, 12, 28, 14, 30152
+idct16_shuff3: dw 1, 17, 3, 19, 5, 21, 7, 23, 9, 25, 11, 27, 13, 29, 15, 31153
+idct16_shuff4: dd 0, 8, 2, 10, 4, 12, 6, 14154
+idct16_shuff5: dd 1, 9, 3, 11, 5, 13, 7, 15155
+156
+157
+tab_AVX512_idct16_1: dw 90, 87, 80, 70, 57, 43, 25, 9, 90, 87, 80, 70, 57, 43, 25, 9, 80, 9, -70, -87, -25, 57, 90, 43, 80, 9, -70, -87, -25, 57, 90, 43158
+ dw 87, 57, 9, -43, -80, -90, -70, -25, 87, 57, 9, -43, -80, -90, -70, -25, 70, -43, -87, 9, 90, 25, -80, -57, 70, -43, -87, 9, 90, 25, -80, -57159
+ dw 57, -80, -25, 90, -9, -87, 43, 70, 57, -80, -25, 90, -9, -87, 43, 70, 25, -70, 90, -80, 43, 9, -57, 87, 25, -70, 90, -80, 43, 9, -57, 87160
+ dw 43, -90, 57, 25, -87, 70, 9, -80, 43, -90, 57, 25, -87, 70, 9, -80, 9, -25, 43, -57, 70, -80, 87, -90, 9, -25, 43, -57, 70, -80, 87, -90161
+162
+tab_AVX512_idct16_2: dw 64, 89, 83, 75, 64, 50, 36, 18, 64, 89, 83, 75, 64, 50, 36, 18, 64, 50, -36, -89, -64, 18, 83, 75, 64, 50, -36, -89, -64, 18, 83, 75163
+ dw 64, 75, 36, -18, -64, -89, -83, -50, 64, 75, 36, -18, -64, -89, -83, -50, 64, 18, -83, -50, 64, 75, -36, -89, 64, 18, -83, -50, 64, 75, -36, -89164
+ dw 64, -18, -83, 50, 64, -75, -36, 89, 64, -18, -83, 50, 64, -75, -36, 89, 64, -75, 36, 18, -64, 89, -83, 50, 64, -75, 36, 18, -64, 89, -83, 50165
+ dw 64, -50, -36, 89, -64, -18, 83, -75, 64, -50, -36, 89, -64, -18, 83, -75, 64, -89, 83, -75, 64, -50, 36, -18, 64, -89, 83, -75, 64, -50, 36, -18166
+167
+idct16_AVX512_shuff: dd 0, 4, 2, 6, 1, 5, 3, 7, 8, 12, 10, 14, 9, 13, 11, 15168
+169
+idct16_AVX512_shuff1: dd 2, 6, 0, 4, 3, 7, 1, 5, 10, 14, 8, 12, 11, 15, 9, 13170
+171
+idct16_AVX512_shuff2: dq 0, 1, 8, 9, 4, 5, 12, 13172
+idct16_AVX512_shuff3: dq 2, 3, 10, 11, 6, 7, 14, 15173
+idct16_AVX512_shuff4: dq 4, 5, 12, 13, 0, 1, 8, 9174
+idct16_AVX512_shuff5: dq 6, 7, 14, 15, 2, 3, 10, 11175
+idct16_AVX512_shuff6: times 4 db 14, 15, 12, 13, 10, 11, 8, 9, 6, 7, 4, 5, 2, 3, 0, 1176
177
tab_idct32_1: dw 90 ,90 ,88 ,85, 82, 78, 73, 67, 61, 54, 46, 38, 31, 22, 13, 4178
dw 90, 82, 67, 46, 22, -4, -31, -54, -73, -85, -90, -88, -78, -61, -38, -13179
180
dw 64, -87, 75, -57, 36, -9, -18, 43, -64, 80, -89, 90, -83, 70, -50, 25181
dw 64, -90, 89, -87, 83, -80, 75, -70, 64, -57, 50, -43, 36, -25, 18, -9182
183
+184
+tab_idct32_AVX512_1: dw 90 ,90 ,88 ,85, 82, 78, 73, 67, 90 ,90 ,88 ,85, 82, 78, 73, 67, 90, 82, 67, 46, 22, -4, -31, -54, 90, 82, 67, 46, 22, -4, -31, -54185
+ dw 61, 54, 46, 38, 31, 22, 13, 4, 61, 54, 46, 38, 31, 22, 13, 4, -73, -85, -90, -88, -78, -61, -38, -13, -73, -85, -90, -88, -78, -61, -38, -13186
+ dw 88, 67, 31, -13, -54, -82, -90, -78, 88, 67, 31, -13, -54, -82, -90, -78, 85, 46, -13, -67, -90, -73, -22, 38, 85, 46, -13, -67, -90, -73, -22, 38187
+ dw -46, -4, 38, 73, 90, 85, 61, 22, -46, -4, 38, 73, 90, 85, 61, 22, 82, 88, 54, -4, -61, -90, -78, -31, 82, 88, 54, -4, -61, -90, -78, -31188
+ dw 82, 22, -54, -90, -61, 13, 78, 85, 82, 22, -54, -90, -61, 13, 78, 85, 78, -4, -82, -73, 13, 85, 67, -22, 78, -4, -82, -73, 13, 85, 67, -22189
+ dw 31, -46, -90, -67, 4, 73, 88, 38, 31, -46, -90, -67, 4, 73, 88, 38, -88, -61, 31, 90, 54, -38, -90, -46, -88, -61, 31, 90, 54, -38, -90, -46190
+ dw 73, -31, -90, -22, 78, 67, -38, -90, 73, -31, -90, -22, 78, 67, -38, -90, 67, -54, -78, 38, 85, -22, -90, 4, 67, -54, -78, 38, 85, -22, -90, 4191
+ dw -13, 82, 61, -46, -88, -4, 85, 54, -13, 82, 61, -46, -88, -4, 85, 54, 90, 13, -88, -31, 82, 46, -73, -61, 90, 13, -88, -31, 82, 46, -73, -61192
+193
+tab_idct32_AVX512_5: dw 4, -13, 22, -31, 38, -46, 54, -61, 4, -13, 22, -31, 38, -46, 54, -61, 13, -38, 61, -78, 88, -90, 85, -73, 13, -38, 61, -78, 88, -90, 85, -73194
+ dw 67, -73, 78, -82, 85, -88, 90, -90, 67, -73, 78, -82, 85, -88, 90, -90, 54, -31, 4, 22, -46, 67, -82, 90, 54, -31, 4, 22, -46, 67, -82, 90195
+ dw 22, -61, 85, -90, 73, -38, -4, 46, 22, -61, 85, -90, 73, -38, -4, 46, 31, -78, 90, -61, 4, 54, -88, 82, 31, -78, 90, -61, 4, 54, -88, 82196
+ dw -78, 90, -82, 54, -13, -31, 67, -88, -78, 90, -82, 54, -13, -31, 67, -88, -38, -22, 73, -90, 67, -13, -46, 85, -38, -22, 73, -90, 67, -13, -46, 85197
+ dw 38, -88, 73, -4, -67, 90, -46, -31, 38, -88, 73, -4, -67, 90, -46, -31, 46, -90, 38, 54, -90, 31, 61, -88, 46, -90, 38, 54, -90, 31, 61, -88198
+ dw 85, -78, 13, 61, -90, 54, 22, -82, 85, -78, 13, 61, -90, 54, 22, -82, 22, 67, -85, 13, 73, -82, 4, 78, 22, 67, -85, 13, 73, -82, 4, 78199
+ dw 54, -85, -4, 88, -46, -61, 82, 13, 54, -85, -4, 88, -46, -61, 82, 13, 61, -73, -46, 82, 31, -88, -13, 90, 61, -73, -46, 82, 31, -88, -13, 90200
+ dw -90, 38, 67, -78, -22, 90, -31, -73, -90, 38, 67, -78, -22, 90, -31, -73, -4, -90, 22, 85, -38, -78, 54, 67, -4, -90, 22, 85, -38, -78, 54, 67201
+202
+203
+tab_idct32_AVX512_2: dw 64, 89, 83, 75, 64, 50, 36, 18, 64, 89, 83, 75, 64, 50, 36, 18, 64, 75, 36, -18, -64, -89, -83, -50, 64, 75, 36, -18, -64, -89, -83, -50204
+ dw 64, 50, -36, -89, -64, 18, 83, 75, 64, 50, -36, -89, -64, 18, 83, 75, 64, 18, -83, -50, 64, 75, -36, -89, 64, 18, -83, -50, 64, 75, -36, -89205
+ dw 64, -18, -83, 50, 64, -75, -36, 89, 64, -18, -83, 50, 64, -75, -36, 89, 64, -50, -36, 89, -64, -18, 83, -75, 64, -50, -36, 89, -64, -18, 83, -75206
+ dw 64, -75, 36, 18, -64, 89, -83, 50, 64, -75, 36, 18, -64, 89, -83, 50, 64, -89, 83, -75, 64, -50, 36, -18, 64, -89, 83, -75, 64, -50, 36, -18207
+208
+tab_idct32_AVX512_3: dw 90, 87, 80, 70, 57, 43, 25, 9, 90, 87, 80, 70, 57, 43, 25, 9, 87, 57, 9, -43, -80, -90, -70, -25, 87, 57, 9, -43, -80, -90, -70, -25209
+ dw 80, 9, -70, -87, -25, 57, 90, 43, 80, 9, -70, -87, -25, 57, 90, 43, 70, -43, -87, 9, 90, 25, -80, -57, 70, -43, -87, 9, 90, 25, -80, -57210
+ dw 57, -80, -25, 90, -9, -87, 43, 70, 57, -80, -25, 90, -9, -87, 43, 70, 43, -90, 57, 25, -87, 70, 9, -80, 43, -90, 57, 25, -87, 70, 9, -80211
+ dw 25, -70, 90, -80, 43, 9, -57, 87, 25, -70, 90, -80, 43, 9, -57, 87, 9, -25, 43, -57, 70, -80, 87, -90, 9, -25, 43, -57, 70, -80, 87, -90212
+213
+tab_idct32_AVX512_4: dw 90 ,90 ,88 ,85, 82, 78, 73, 67, 61, 54, 46, 38, 31, 22, 13, 4, 90 ,90 ,88 ,85, 82, 78, 73, 67, 61, 54, 46, 38, 31, 22, 13, 4214
+ dw 90, 82, 67, 46, 22, -4, -31, -54, -73, -85, -90, -88, -78, -61, -38, -13, 90, 82, 67, 46, 22, -4, -31, -54, -73, -85, -90, -88, -78, -61, -38, -13215
+ dw 88, 67, 31, -13, -54, -82, -90, -78, -46, -4, 38, 73, 90, 85, 61, 22, 88, 67, 31, -13, -54, -82, -90, -78, -46, -4, 38, 73, 90, 85, 61, 22216
+ dw 85, 46, -13, -67, -90, -73, -22, 38, 82, 88, 54, -4, -61, -90, -78, -31, 85, 46, -13, -67, -90, -73, -22, 38, 82, 88, 54, -4, -61, -90, -78, -31217
+ dw 82, 22, -54, -90, -61, 13, 78, 85, 31, -46, -90, -67, 4, 73, 88, 38, 82, 22, -54, -90, -61, 13, 78, 85, 31, -46, -90, -67, 4, 73, 88, 38218
+ dw 78, -4, -82, -73, 13, 85, 67, -22, -88, -61, 31, 90, 54, -38, -90, -46, 78, -4, -82, -73, 13, 85, 67, -22, -88, -61, 31, 90, 54, -38, -90, -46219
+ dw 73, -31, -90, -22, 78, 67, -38, -90, -13, 82, 61, -46, -88, -4, 85, 54, 73, -31, -90, -22, 78, 67, -38, -90, -13, 82, 61, -46, -88, -4, 85, 54220
+ dw 67, -54, -78, 38, 85, -22, -90, 4, 90, 13, -88, -31, 82, 46, -73, -61, 67, -54, -78, 38, 85, -22, -90, 4, 90, 13, -88, -31, 82, 46, -73, -61221
+ dw 61, -73, -46, 82, 31, -88, -13, 90, -4, -90, 22, 85, -38, -78, 54, 67, 61, -73, -46, 82, 31, -88, -13, 90, -4, -90, 22, 85, -38, -78, 54, 67222
+ dw 54, -85, -4, 88, -46, -61, 82, 13, -90, 38, 67, -78, -22, 90, -31, -73, 54, -85, -4, 88, -46, -61, 82, 13, -90, 38, 67, -78, -22, 90, -31, -73223
+ dw 46, -90, 38, 54, -90, 31, 61, -88, 22, 67, -85, 13, 73, -82, 4, 78, 46, -90, 38, 54, -90, 31, 61, -88, 22, 67, -85, 13, 73, -82, 4, 78224
+ dw 38, -88, 73, -4, -67, 90, -46, -31, 85, -78, 13, 61, -90, 54, 22, -82, 38, -88, 73, -4, -67, 90, -46, -31, 85, -78, 13, 61, -90, 54, 22, -82225
+ dw 31, -78, 90, -61, 4, 54, -88, 82, -38, -22, 73, -90, 67, -13, -46, 85, 31, -78, 90, -61, 4, 54, -88, 82, -38, -22, 73, -90, 67, -13, -46, 85226
+ dw 22, -61, 85, -90, 73, -38, -4, 46, -78, 90, -82, 54, -13, -31, 67, -88, 22, -61, 85, -90, 73, -38, -4, 46, -78, 90, -82, 54, -13, -31, 67, -88227
+ dw 13, -38, 61, -78, 88, -90, 85, -73, 54, -31, 4, 22, -46, 67, -82, 90, 13, -38, 61, -78, 88, -90, 85, -73, 54, -31, 4, 22, -46, 67, -82, 90228
+ dw 4, -13, 22, -31, 38, -46, 54, -61, 67, -73, 78, -82, 85, -88, 90, -90, 4, -13, 22, -31, 38, -46, 54, -61, 67, -73, 78, -82, 85, -88, 90, -90229
+230
+tab_idct32_AVX512_6: dw 64, 90, 89, 87, 83, 80, 75, 70, 64, 57, 50, 43, 36, 25, 18, 9, 64, 90, 89, 87, 83, 80, 75, 70, 64, 57, 50, 43, 36, 25, 18, 9231
+ dw 64, 87, 75, 57, 36, 9, -18, -43, -64, -80, -89, -90, -83, -70, -50, -25, 64, 87, 75, 57, 36, 9, -18, -43, -64, -80, -89, -90, -83, -70, -50, -25232
+ dw 64, 80, 50, 9, -36, -70, -89, -87, -64, -25, 18, 57, 83, 90, 75, 43, 64, 80, 50, 9, -36, -70, -89, -87, -64, -25, 18, 57, 83, 90, 75, 43233
+ dw 64, 70, 18, -43, -83, -87, -50, 9, 64, 90, 75, 25, -36, -80, -89, -57, 64, 70, 18, -43, -83, -87, -50, 9, 64, 90, 75, 25, -36, -80, -89, -57234
+ dw 64, 57, -18, -80, -83, -25, 50, 90, 64, -9, -75, -87, -36, 43, 89, 70, 64, 57, -18, -80, -83, -25, 50, 90, 64, -9, -75, -87, -36, 43, 89, 70235
+ dw 64, 43, -50, -90, -36, 57, 89, 25, -64, -87, -18, 70, 83, 9, -75, -80, 64, 43, -50, -90, -36, 57, 89, 25, -64, -87, -18, 70, 83, 9, -75, -80236
+ dw 64, 25, -75, -70, 36, 90, 18, -80, -64, 43, 89, 9, -83, -57, 50, 87, 64, 25, -75, -70, 36, 90, 18, -80, -64, 43, 89, 9, -83, -57, 50, 87237
+ dw 64, 9, -89, -25, 83, 43, -75, -57, 64, 70, -50, -80, 36, 87, -18, -90, 64, 9, -89, -25, 83, 43, -75, -57, 64, 70, -50, -80, 36, 87, -18, -90238
+ dw 64, -9, -89, 25, 83, -43, -75, 57, 64, -70, -50, 80, 36, -87, -18, 90, 64, -9, -89, 25, 83, -43, -75, 57, 64, -70, -50, 80, 36, -87, -18, 90239
+ dw 64, -25, -75, 70, 36, -90, 18, 80, -64, -43, 89, -9, -83, 57, 50, -87, 64, -25, -75, 70, 36, -90, 18, 80, -64, -43, 89, -9, -83, 57, 50, -87240
+ dw 64, -43, -50, 90, -36, -57, 89, -25, -64, 87, -18, -70, 83, -9, -75, 80, 64, -43, -50, 90, -36, -57, 89, -25, -64, 87, -18, -70, 83, -9, -75, 80241
+ dw 64, -57, -18, 80, -83, 25, 50, -90, 64, 9, -75, 87, -36, -43, 89, -70, 64, -57, -18, 80, -83, 25, 50, -90, 64, 9, -75, 87, -36, -43, 89, -70242
+ dw 64, -70, 18, 43, -83, 87, -50, -9, 64, -90, 75, -25, -36, 80, -89, 57, 64, -70, 18, 43, -83, 87, -50, -9, 64, -90, 75, -25, -36, 80, -89, 57243
+ dw 64, -80, 50, -9, -36, 70, -89, 87, -64, 25, 18, -57, 83, -90, 75, -43, 64, -80, 50, -9, -36, 70, -89, 87, -64, 25, 18, -57, 83, -90, 75, -43244
+ dw 64, -87, 75, -57, 36, -9, -18, 43, -64, 80, -89, 90, -83, 70, -50, 25, 64, -87, 75, -57, 36, -9, -18, 43, -64, 80, -89, 90, -83, 70, -50, 25245
+ dw 64, -90, 89, -87, 83, -80, 75, -70, 64, -57, 50, -43, 36, -25, 18, -9, 64, -90, 89, -87, 83, -80, 75, -70, 64, -57, 50, -43, 36, -25, 18, -9246
+247
+248
avx2_dct4: dw 64, 64, 64, 64, 64, 64, 64, 64, 64, -64, 64, -64, 64, -64, 64, -64249
dw 83, 36, 83, 36, 83, 36, 83, 36, 36, -83, 36, -83, 36, -83, 36, -83250
251
252
253
tab_idct8_2: times 1 dw 89, 75, 50, 18, 75, -18, -89, -50254
times 1 dw 50, -89, 18, 75, 18, -50, 75, -89255
-256
pb_idct8odd: db 2, 3, 6, 7, 10, 11, 14, 15, 2, 3, 6, 7, 10, 11, 14, 15257
258
+;Scale bits table for rdoQuant259
+tab_nonpsyRdo8 : dq 5, 7, 9, 11260
+tab_nonpsyRdo10: dq 9, 11, 13, 15261
+tab_nonpsyRdo12: dq 13, 15, 17, 19262
+263
SECTION .text264
cextern pd_1265
cextern pd_2266
267
%define DST4_ROUND 16268
%define DCT8_SHIFT1 6269
%define DCT8_ROUND1 32270
+ %define RDO_MAX_4 3271
+ %define RDO_MAX_8 1272
+ %define RDO_MAX_16 0273
+ %define RDO_MAX_32 0274
%elif BIT_DEPTH == 10275
%define DCT4_SHIFT 3276
%define DCT4_ROUND 4277
278
%define DST4_ROUND 4279
%define DCT8_SHIFT1 4280
%define DCT8_ROUND1 8281
+ %define RDO_MAX_4 7282
+ %define RDO_MAX_8 5283
+ %define RDO_MAX_16 3284
+ %define RDO_MAX_32 1285
%elif BIT_DEPTH == 8286
%define DCT4_SHIFT 1287
%define DCT4_ROUND 1288
289
%define DST4_ROUND 1290
%define DCT8_SHIFT1 2291
%define DCT8_ROUND1 2292
+ %define RDO_MAX_4 11293
+ %define RDO_MAX_8 9294
+ %define RDO_MAX_16 7295
+ %define RDO_MAX_32 5296
%else297
%error Unsupported BIT_DEPTH!298
%endif299
300
dec r3d301
jnz .loop302
RET303
+%if ARCH_X86_64 == 1304
+INIT_ZMM avx512305
+cglobal denoise_dct, 4, 4, 22306
+ pxor m16, m16307
+ sub r3d, 16308
+ je .coeff16309
+ add r3d, 16310
+ shr r3d, 5311
+ jmp .loop312
+313
+.coeff16:314
+ movu ym19, [r0]315
+ pabsw ym17, ym19316
+ movu m2, [r1]317
+ pmovsxwd m18, ym17318
+ paddd m2, m18319
+ movu [r1], m2320
+ movu ym3, [r2]321
+ psubusw ym17, ym3322
+ pcmpgtw ym18, ym17, ym16323
+ pand ym17, ym18324
+ psignw ym17, ym19325
+ movu [r0], ym17326
+ RET327
+328
+.loop:329
+ movu m21, [r0]330
+ pabsw m17, m21331
+ movu m2, [r1]332
+ pmovsxwd m4, ym17333
+ paddd m2, m4334
+ movu [r1], m2335
+ vextracti64x4 ym4, m17, 1336
+337
+ movu m2, [r1 + mmsize]338
+ pmovsxwd m3, ym4339
+ paddd m2, m3340
+ movu [r1 + mmsize], m2341
+ movu m3, [r2]342
+ psubusw m17, m3343
+344
+ vextracti64x4 ym20, m17, 1345
+ pcmpgtw ym18, ym17, ym16346
+ pcmpgtw ym19, ym20, ym16347
+ vinserti64x4 m18, m18, ym19, 1348
+349
+ pand m17, m18350
+ vextracti64x4 ym19, m17, 1351
+ vextracti64x4 ym20, m21, 1352
+ psignw ym17, ym21353
+ psignw ym19, ym20354
+ vinserti64x4 m17, m17, ym19, 1355
+356
+ movu [r0], m17357
+ add r0, mmsize358
+ add r1, mmsize * 2359
+ add r2, mmsize360
+ dec r3d361
+ jnz .loop362
+ RET363
+%endif ; ARCH_X86_64 == 1364
365
%if ARCH_X86_64 == 1366
%macro DCT8_PASS_1 4367
368
movu [r1 + 96], m10369
RET370
371
+372
+%macro DCT8_AVX512_PASS_1 4373
+ vpmaddwd m%2, m3, m%1374
+ vpsrlq m8, m%2, 32375
+ vpaddd m%2, m8376
+ vpaddd m%2, m5377
+ vpsrad m%2, DCT8_SHIFT1378
+379
+ vpmaddwd m%4, m2, m%3380
+ vpsrlq m8, m%4, 32381
+ vpaddd m%4, m8382
+ vpaddd m%4, m5383
+ vpsrad m%4, DCT8_SHIFT1384
+385
+ vpackssdw m%2, m%4386
+ vpermw m%2, m1, m%2387
+%endmacro388
+389
+%macro DCT8_AVX512_PASS_2 4390
+ vpmaddwd m0, m9, m%1391
+ vpmaddwd m1, m10, m%1392
+ vpsrldq m2, m0, 8393
+ vpsrldq m3, m1, 8394
+ vpaddd m0, m2395
+ vpaddd m1, m3396
+ vpsrlq m2, m0, 32397
+ vpsrlq m3, m1, 32398
+ vpaddd m0, m2399
+ vpaddd m1, m3400
+ vpaddd m0, m5401
+ vpsrad m0, DCT8_SHIFT2402
+ vpaddd m1, m5403
+ vpsrad m1, DCT8_SHIFT2404
+ vpackssdw m0, m1405
+ vpermw m0, m19, m0406
+407
+ vpmaddwd m1, m9, m%2408
+ vpmaddwd m2, m10, m%2409
+ vpsrldq m3, m1, 8410
+ vpsrldq m4, m2, 8411
+ vpaddd m1, m3412
+ vpaddd m2, m4413
+ vpsrlq m3, m1, 32414
+ vpsrlq m4, m2, 32415
+ vpaddd m1, m3416
+ vpaddd m2, m4417
+ vpaddd m1, m5418
+ vpsrad m1, DCT8_SHIFT2419
+ vpaddd m2, m5420
+ vpsrad m2, DCT8_SHIFT2421
+ vpackssdw m1, m2422
+ vpermw m1, m19, m1423
+ vinserti128 ym0, ym0, xm1, 1424
+425
+ vpmaddwd m1, m9, m%3426
+ vpmaddwd m2, m10, m%3427
+ vpsrldq m3, m1, 8428
+ vpsrldq m4, m2, 8429
+ vpaddd m1, m3430
+ vpaddd m2, m4431
+ vpsrlq m3, m1, 32432
+ vpsrlq m4, m2, 32433
+ vpaddd m1, m3434
+ vpaddd m2, m4435
+ vpaddd m1, m5436
+ vpsrad m1, DCT8_SHIFT2437
+ vpaddd m2, m5438
+ vpsrad m2, DCT8_SHIFT2439
+ vpackssdw m1, m2440
+ vpermw m1, m19, m1441
+442
+ vpmaddwd m2, m9, m%4443
+ vpmaddwd m3, m10, m%4444
+ vpsrldq m4, m2, 8445
+ vpsrldq m6, m3, 8446
+ vpaddd m2, m4447
+ vpaddd m3, m6448
+ vpsrlq m4, m2, 32449
+ vpsrlq m6, m3, 32450
+ vpaddd m2, m4451
+ vpaddd m3, m6452
+ vpaddd m2, m5453
+ vpsrad m2, DCT8_SHIFT2454
+ vpaddd m3, m5455
+ vpsrad m3, DCT8_SHIFT2456
+ vpackssdw m2, m3457
+ vpermw m2, m19, m2458
+459
+ vinserti128 ym1, ym1, xm2, 1460
+ vinserti64x4 m0, m0, ym1, 1461
+%endmacro462
+463
+INIT_ZMM avx512464
+cglobal dct8, 3, 7, 24465
+466
+ vbroadcasti32x4 m5, [pd_ %+ DCT8_ROUND1]467
+ vbroadcasti32x8 m4, [dct8_shuf]468
+ vbroadcasti32x4 m19, [dct8_shuf9_AVX512]469
+470
+ add r2d, r2d471
+ lea r3, [r2 * 3]472
+ lea r4, [r0 + r2 * 4]473
+ lea r5, [tab_dct8]474
+ lea r6, [tab_dct8_avx512]475
+476
+ ;pass1477
+ mova xm0, [r0]478
+ vinserti128 ym0, ym0, [r4], 1479
+ mova xm1, [r0 + r2]480
+ vinserti128 ym1, ym1, [r4 + r2], 1481
+ mova xm2, [r0 + r2 * 2]482
+ vinserti128 ym2, ym2, [r4 + r2 * 2], 1483
+ mova xm3, [r0 + r3]484
+ vinserti128 ym3, ym3, [r4 + r3], 1485
+486
+ vinserti64x4 m0, m0, ym2, 1487
+ vinserti64x4 m1, m1, ym3, 1488
+489
+ vpunpcklqdq m2, m0, m1490
+ vpunpckhqdq m0, m1491
+492
+ vpshufb m0, m4493
+ vpaddw m3, m2, m0494
+ vpsubw m2, m0495
+496
+ vbroadcasti32x8 m1, [dct8_shuf7_AVX512]497
+498
+ ; Load all the coefficients togather for better caching499
+ vpbroadcastq m20, [r6 + 0 * 8]500
+ vpbroadcastq m21, [r6 + 1 * 8]501
+ vpbroadcastq m22, [r6 + 2 * 8]502
+ vpbroadcastq m23, [r6 + 3 * 8]503
+ vpbroadcastq m7, [r6 + 4 * 8]504
+ vpbroadcastq m12, [r6 + 5 * 8]505
+ vpbroadcastq m14, [r6 + 6 * 8]506
+ vpbroadcastq m16, [r6 + 7 * 8]507
+508
+ DCT8_AVX512_PASS_1 20, 9, 21, 10509
+ DCT8_AVX512_PASS_1 22, 11, 23, 10510
+ DCT8_AVX512_PASS_1 7, 13, 12, 10511
+ DCT8_AVX512_PASS_1 14, 15, 16, 10512
+513
+ ;pass2514
+ vbroadcasti32x4 m5, [pd_ %+ DCT8_ROUND2]515
+516
+ vinserti64x4 m9, m9, ym11, 1517
+ vinserti64x4 m10, m13, ym15, 1518
+519
+ ;Load all the coefficients togather for better caching and reuse common coefficients from PASS 1520
+ vbroadcasti32x4 m21, [r5 + 1 * 16]521
+ vbroadcasti32x4 m22, [r5 + 2 * 16]522
+ vbroadcasti32x4 m23, [r5 + 3 * 16]523
+ vbroadcasti32x4 m12, [r5 + 5 * 16]524
+ vbroadcasti32x4 m14, [r5 + 6 * 16]525
+ vbroadcasti32x4 m16, [r5 + 7 * 16]526
+527
+ DCT8_AVX512_PASS_2 20, 21, 22, 23528
+ movu [r1], m0529
+ DCT8_AVX512_PASS_2 7, 12, 14, 16530
+ movu [r1 + 64], m0531
+ RET532
+533
%macro DCT16_PASS_1_E 2534
vpbroadcastq m7, [r7 + %1]535
536
537
dec r4d538
jnz .pass2539
RET540
+%macro DCT16_avx512_PASS_1_O 4541
+ vbroadcasti32x4 m1, [r5 + %1]542
+543
+ pmaddwd m3, m6, m1544
+ vpsrldq m11, m3, 8545
+ vpaddd m3, m11546
+547
+ pmaddwd m11, m8, m1548
+ vpsrldq m12, m11, 8549
+ vpaddd m11, m12550
+551
+ vpunpcklqdq m12, m3, m11552
+ vpsrldq m11, m12, 4553
+ vpaddd m11, m12554
+555
+ pmaddwd m3, m10, m1556
+ vpsrldq m12, m3, 8557
+ vpaddd m3, m12558
+559
+ pmaddwd m12, m2, m1560
+ vpsrldq m13, m12, 8561
+ vpaddd m12, m13562
+563
+ vpunpcklqdq m13, m3, m12564
+ vpsrldq m12, m13, 4565
+ vpaddd m12, m13566
+567
+ mova m%3, m26568
+ vpermi2d m%3, m11, m12569
+ paddd m%3, m0570
+ psrad m%3, DCT_SHIFT571
+572
+ ; next row start573
+ vbroadcasti32x4 m1, [r5 + %2]574
+575
+ pmaddwd m3, m6, m1576
+ vpsrldq m11, m3, 8577
+ vpaddd m3, m11578
+579
+ pmaddwd m11, m8, m1580
+ vpsrldq m12, m11, 8581
+ vpaddd m11, m12582
+583
+ vpunpcklqdq m12, m3, m11584
+ vpsrldq m11, m12, 4585
+ vpaddd m11, m12586
+587
+ pmaddwd m3, m10, m1588
+ vpsrldq m12, m3, 8589
+ vpaddd m3, m12590
+591
+ pmaddwd m12, m2, m1592
+ vpsrldq m13, m12, 8593
+ vpaddd m12, m13594
+595
+ vpunpcklqdq m13, m3, m12596
+ vpsrldq m12, m13, 4597
+ vpaddd m12, m13598
+599
+ mova m%4, m26600
+ vpermi2d m%4, m11, m12601
+ paddd m%4, m0602
+ psrad m%4, DCT_SHIFT603
+ ;next row end604
+605
+ packssdw m%3, m%4606
+ vpermw m%4, m25, m%3607
+%endmacro608
+609
+%macro DCT16_AVX512_PASS_1_LOOP 0610
+ vbroadcasti32x8 m1, [dct16_shuf1]611
+ mova m2, [dct16_shuf3_AVX512]612
+ mova m3, [dct16_shuf4_AVX512]613
+614
+ movu ym4, [r0]615
+ movu ym5, [r0 + r2]616
+ vinserti64x4 m4, m4, ym5, 1617
+618
+ movu ym5, [r0 + 2 * r2]619
+ movu ym6, [r0 + r3]620
+ vinserti64x4 m5, m5, ym6, 1621
+622
+ mova m6, m2623
+ mova m7, m3624
+ vpermi2q m6, m4, m5625
+ vpermi2q m7, m4, m5626
+627
+ movu ym4, [r4]628
+ movu ym5, [r4 + r2]629
+ vinserti64x4 m4, m4, ym5, 1630
+631
+ movu ym5, [r4 + 2 * r2]632
+ movu ym8, [r4 + r3]633
+ vinserti64x4 m5, m5, ym8, 1634
+635
+ mova m8, m2636
+ mova m9, m3637
+ vpermi2q m8, m4, m5638
+ vpermi2q m9, m4, m5639
+640
+ vpshufb m7, m1641
+ vpshufb m9, m1642
+643
+ paddw m4, m6, m7644
+ psubw m6, m7645
+646
+ paddw m5, m8, m9647
+ psubw m8, m9648
+649
+ lea r0, [r0 + 8 * r2]650
+ lea r4, [r0 + r2 * 4]651
+652
+ movu ym7, [r0]653
+ movu ym9, [r0 + r2]654
+ vinserti64x4 m7, m7, ym9, 1655
+656
+ movu ym9, [r0 + 2 * r2]657
+ movu ym10, [r0 + r3]658
+ vinserti64x4 m9, m9, ym10, 1659
+660
+ mova m10, m2661
+ mova m11, m3662
+ vpermi2q m10, m7, m9663
+ vpermi2q m11, m7, m9664
+665
+ vpshufb m11, m1666
+ paddw m7, m10, m11667
+ psubw m10, m11668
+669
+ movu ym9, [r4]670
+ movu ym11, [r4 + r2]671
+ vinserti64x4 m9, m9, ym11, 1672
+673
+ movu ym11, [r4 + 2 * r2]674
+ movu ym12, [r4 + r3]675
+ vinserti64x4 m11, m11, ym12, 1676
+677
+ vpermi2q m2, m9, m11678
+ vpermi2q m3, m9, m11679
+680
+ vpshufb m3, m1681
+ paddw m9, m2, m3682
+ psubw m2, m3683
+%endmacro684
+685
+%macro DCT16_avx512_PASS_1_E 4686
+ vpbroadcastq m1, [r5 + %1]687
+688
+ pmaddwd m19, m11, m1689
+ vpsrldq m12, m19, 4690
+ vpaddd m12, m19691
+692
+ pmaddwd m19, m13, m1693
+ vpsrldq m18, m19, 4694
+ vpaddd m18, m19695
+696
+ mova m%2, m27697
+ vpermi2d m%2, m12, m18698
+ paddd m%2, m0699
+ psrad m%2, DCT_SHIFT700
+701
+ ; 2nd row702
+ vpbroadcastq m1, [r5 + %3]703
+704
+ pmaddwd m19, m11, m1705
+ vpsrldq m12, m19, 4706
+ vpaddd m12, m19707
+708
+ pmaddwd m19, m13, m1709
+ vpsrldq m18, m19, 4710
+ vpaddd m18, m19711
+712
+ mova m%4, m27713
+ vpermi2d m%4, m12, m18714
+ paddd m%4, m0715
+ psrad m%4, DCT_SHIFT716
+717
+ packssdw m%2, m%4718
+ vpermw m%4, m25, m%2719
+%endmacro720
+721
+%macro DCT16_PASS2_AVX512 10722
+ vpmaddwd m5, m%2, m%1723
+ vpsrldq m6, m5, 8724
+ vpaddd m5, m6725
+ vpsrldq m6, m5, 4726
+ vpaddd m5, m6727
+728
+ vpmaddwd m6, m%3, m%1729
+ vpsrldq m7, m6, 8730
+ vpaddd m6, m7731
+ vpsrldq m7, m6, 4732
+ vpaddd m6, m7733
+ vpunpckldq m7, m5, m6734
+735
+ vpmaddwd m5, m%4, m%1736
+ vpsrldq m6, m5, 8737
+ vpaddd m5, m6738
+ vpsrldq m6, m5, 4739
+ vpaddd m5, m6740
+741
+ vpmaddwd m6, m%5, m%1742
+ vpsrldq m8, m6, 8743
+ vpaddd m6, m8744
+ vpsrldq m8, m6, 4745
+ vpaddd m6, m8746
+ vpunpckldq m8, m5, m6747
+748
+ vpunpcklqdq m5, m7, m8749
+ vpermd m5, m2, m5750
+ vpsrldq m6, m5, 4751
+ vpaddd m5, m6752
+753
+ vpmaddwd m6, m%6, m%1754
+ vpsrldq m7, m6, 8755
+ vpaddd m6, m7756
+ vpsrldq m7, m6, 4757
+ vpaddd m6, m7758
+759
+ vpmaddwd m7, m%7, m%1760
+ vpsrldq m8, m7, 8761
+ vpaddd m7, m8762
+ vpsrldq m8, m7, 4763
+ vpaddd m7, m8764
+ vpunpckldq m8, m6, m7765
+766
+ vpmaddwd m6, m%8, m%1767
+ vpsrldq m7, m6, 8768
+ vpaddd m6, m7769
+ vpsrldq m7, m6, 4770
+ vpaddd m6, m7771
+772
+ vpmaddwd m7, m%9, m%1773
+ vpsrldq m4, m7, 8774
+ vpaddd m7, m4775
+ vpsrldq m4, m7, 4776
+ vpaddd m7, m4777
+ vpunpckldq m4, m6, m7778
+779
+ vpunpcklqdq m6, m8, m4780
+ vpermd m6, m2, m6781
+ vpsrldq m7, m6, 4782
+ vpaddd m6, m7783
+784
+ paddd m5, m0785
+ psrad m5, DCT_SHIFT2786
+ paddd m6, m0787
+ psrad m6, DCT_SHIFT2788
+789
+ packssdw m5, m6790
+ vpermw m%10, m3, m5791
+%endmacro792
+793
+INIT_ZMM avx512794
+cglobal dct16, 3, 6, 29795
+796
+%if BIT_DEPTH == 12797
+ %define DCT_SHIFT 7798
+ vbroadcasti32x4 m0, [pd_64]799
+%elif BIT_DEPTH == 10800
+ %define DCT_SHIFT 5801
+ vbroadcasti32x4 m0, [pd_16]802
+%elif BIT_DEPTH == 8803
+ %define DCT_SHIFT 3804
+ vbroadcasti32x4 m0, [pd_4]805
+%else806
+ %error Unsupported BIT_DEPTH!807
+%endif808
+%define DCT_SHIFT2 10809
+810
+ add r2d, r2d811
+ lea r3, [r2 * 3]812
+ lea r4, [r0 + r2 * 4]813
+ lea r5, [tab_dct16_1 + 8 * 16]814
+815
+ ;Load reuseable table once to save memory movments816
+ mova m25, [dct16_shuf5_AVX512]817
+ mova m26, [dct16_shuf2_AVX512]818
+ mova m27, [dct16_shuf7_AVX512]819
+ vbroadcasti32x8 m28, [dct16_shuf6_AVX512]820
+821
+ DCT16_AVX512_PASS_1_LOOP822
+ DCT16_avx512_PASS_1_O -7 * 16, -5 * 16, 15, 14 ;row 1, 3823
+ DCT16_avx512_PASS_1_O -3 * 16, -1 * 16, 16, 15 ;row 5, 7824
+ DCT16_avx512_PASS_1_O 1 * 16, 3 * 16, 17, 16 ;row 9, 11825
+ DCT16_avx512_PASS_1_O 5 * 16, 7 * 16, 18, 17 ;row 13, 15826
+827
+ vbroadcasti32x8 m1, [dct16_shuf2]828
+ pshufb m4, m1829
+ pshufb m5, m1830
+ pshufb m7, m1831
+ pshufb m9, m1832
+833
+ vpsrldq m3, m4, 2834
+ vpsubw m11, m4, m3835
+ vpsrldq m6, m5, 2836
+ vpsubw m12, m5, m6837
+ vpsrldq m8, m7, 2838
+ vpsubw m13, m7, m8839
+ vpsrldq m10, m9, 2840
+ vpsubw m18, m9, m10841
+842
+ vpermw m11, m28, m11843
+ vpermw m12, m28, m12844
+ vinserti64x4 m11, m11, ym12, 1845
+846
+ vpermw m13, m28, m13847
+ vpermw m18, m28, m18848
+ vinserti64x4 m13, m13, ym18, 1849
+850
+ DCT16_avx512_PASS_1_E -6 * 16, 21, -2 * 16, 20 ; row 2, 6851
+ DCT16_avx512_PASS_1_E 2 * 16, 22, 6 * 16, 21 ; row 10, 14852
+853
+ vpaddw m11, m4, m3854
+ vpaddw m12, m5, m6855
+ vpaddw m13, m7, m8856
+ vpaddw m18, m9, m10857
+858
+ vpermw m11, m28, m11859
+ vpermw m12, m28, m12860
+ vinserti64x4 m11, m11, ym12, 1861
+862
+ vpermw m13, m28, m13863
+ vpermw m18, m28, m18864
+ vinserti64x4 m13, m13, ym18, 1865
+866
+ DCT16_avx512_PASS_1_E -8 * 16, 23, 0 * 16, 22 ; row 0, 8867
+ DCT16_avx512_PASS_1_E -4 * 16, 24, 4 * 16, 23 ; row 4, 12868
+869
+ ;PASS2870
+ vbroadcasti128 m0, [pd_512]871
+872
+ lea r5, [tab_dct16]873
+ mova m2, [dct16_shuf9_AVX512]874
+ vbroadcasti32x8 m3, [dct16_shuf8_AVX512]875
+876
+ vbroadcasti32x8 m1, [r5 + 0 * 32]877
+ DCT16_PASS2_AVX512 1, 14, 15, 16, 17, 20, 21, 22, 23, 9878
+ vbroadcasti32x8 m1, [r5 + 1 * 32]879
+ DCT16_PASS2_AVX512 1, 14, 15, 16, 17, 20, 21, 22, 23, 10880
+ vinserti64x4 m9, m9, ym10, 1881
+ movu [r1 + 0 * 64], m9882
+883
+ vbroadcasti32x8 m1, [r5 + 2 * 32]884
+ DCT16_PASS2_AVX512 1, 14, 15, 16, 17, 20, 21, 22, 23, 9885
+ vbroadcasti32x8 m1, [r5 + 3 * 32]886
+ DCT16_PASS2_AVX512 1, 14, 15, 16, 17, 20, 21, 22, 23, 10887
+ vinserti64x4 m9, m9, ym10, 1888
+ movu [r1 + 1 * 64], m9889
+890
+ vbroadcasti32x8 m1, [r5 + 4 * 32]891
+ DCT16_PASS2_AVX512 1, 14, 15, 16, 17, 20, 21, 22, 23, 9892
+ vbroadcasti32x8 m1, [r5 + 5 * 32]893
+ DCT16_PASS2_AVX512 1, 14, 15, 16, 17, 20, 21, 22, 23, 10894
+ vinserti64x4 m9, m9, ym10, 1895
+ movu [r1 + 2 * 64], m9896
+897
+ vbroadcasti32x8 m1, [r5 + 6 * 32]898
+ DCT16_PASS2_AVX512 1, 14, 15, 16, 17, 20, 21, 22, 23, 9899
+ vbroadcasti32x8 m1, [r5 + 7 * 32]900
+ DCT16_PASS2_AVX512 1, 14, 15, 16, 17, 20, 21, 22, 23, 10901
+ vinserti64x4 m9, m9, ym10, 1902
+ movu [r1 + 3 * 64], m9903
+904
+ vbroadcasti32x8 m1, [r5 + 8 * 32]905
+ DCT16_PASS2_AVX512 1, 14, 15, 16, 17, 20, 21, 22, 23, 9906
+ vbroadcasti32x8 m1, [r5 + 9 * 32]907
+ DCT16_PASS2_AVX512 1, 14, 15, 16, 17, 20, 21, 22, 23, 10908
+ vinserti64x4 m9, m9, ym10, 1909
+ movu [r1 + 4 * 64], m9910
+911
+ vbroadcasti32x8 m1, [r5 + 10 * 32]912
+ DCT16_PASS2_AVX512 1, 14, 15, 16, 17, 20, 21, 22, 23, 9913
+ vbroadcasti32x8 m1, [r5 + 11 * 32]914
+ DCT16_PASS2_AVX512 1, 14, 15, 16, 17, 20, 21, 22, 23, 10915
+ vinserti64x4 m9, m9, ym10, 1916
+ movu [r1 + 5 * 64], m9917
+918
+ vbroadcasti32x8 m1, [r5 + 12 * 32]919
+ DCT16_PASS2_AVX512 1, 14, 15, 16, 17, 20, 21, 22, 23, 9920
+ vbroadcasti32x8 m1, [r5 + 13 * 32]921
+ DCT16_PASS2_AVX512 1, 14, 15, 16, 17, 20, 21, 22, 23, 10922
+ vinserti64x4 m9, m9, ym10, 1923
+ movu [r1 + 6 * 64], m9924
+925
+ vbroadcasti32x8 m1, [r5 + 14 * 32]926
+ DCT16_PASS2_AVX512 1, 14, 15, 16, 17, 20, 21, 22, 23, 9927
+ vbroadcasti32x8 m1, [r5 + 15 * 32]928
+ DCT16_PASS2_AVX512 1, 14, 15, 16, 17, 20, 21, 22, 23, 10929
+ vinserti64x4 m9, m9, ym10, 1930
+ movu [r1 + 7 * 64], m9931
+ RET932
933
%macro DCT32_PASS_1 4934
vbroadcasti128 m8, [r7 + %1]935
-936
pmaddwd m11, m%3, m8937
pmaddwd m12, m%4, m8938
phaddd m11, m12939
940
jnz .pass2941
RET942
943
+944
+%macro DCT32_avx512_LOOP 4945
+ movu m1, [r0]946
+ movu m2, [r0 + r2]947
+948
+ vinserti64x4 m3, m1, ym2, 1 ; row 0l, 1l949
+ vextracti64x4 ym4, m1, 1950
+ vinserti64x4 m2, m2, ym4, 0 ; row 0h, 1h951
+ vpermw m2, m31, m2952
+953
+ psubw m%1, m3, m2 ; O954
+ paddw m3, m2 ; E955
+ mova [r9 + %3 * 64], m3956
+957
+ movu m1, [r0 + 2 * r2]958
+ movu m5, [r0 + r3]959
+960
+ vinserti64x4 m6, m1, ym5, 1 ; row 2l, 3l961
+ vextracti64x4 ym7, m1, 1962
+ vinserti64x4 m5, m5, ym7, 0 ; row 2h, 3h963
+ vpermw m5, m31, m5964
+965
+ psubw m%2, m6, m5 ; O966
+ paddw m6, m5 ; E967
+ mova [r9 + %4 * 64], m6968
+%endmacro969
+970
+%macro DCT32_avx512_PASS_1_O 3971
+ pmaddwd m10, m%2, m9972
+ vpsrldq m11, m10, 8973
+ vpaddd m10, m11974
+975
+ pmaddwd m11, m%3, m9976
+ vpsrldq m12, m11, 8977
+ vpaddd m11, m12978
+979
+ mova m12, m8980
+ vpermi2d m12, m10, m11981
+ vpsrldq m10, m12, 8982
+ vpaddd m12, m10983
+ vpsrldq m10, m12, 4984
+ vpaddd m12, m10985
+986
+ vpaddd m12, m0987
+ vpsrad m12, DCT_SHIFT988
+ vpackssdw m12, m12989
+ vpermw m12, m30, m12990
+ movq [r5 + %1], xm12991
+%endmacro992
+993
+%macro DCT32_avx512_PASS_1_ROW_O 0994
+ vbroadcasti32x8 m9, [r7 + 1 * 32]995
+996
+ DCT32_avx512_LOOP 13, 14, 0, 1997
+ DCT32_avx512_PASS_1_O 1 * 64 + 0 * 8, 13, 14998
+999
+ lea r0, [r0 + 4 * r2]1000
+ DCT32_avx512_LOOP 15, 16, 2, 31001
+ DCT32_avx512_PASS_1_O 1 * 64 + 1 * 8, 15, 161002
+1003
+ lea r0, [r0 + 4 * r2]1004
+ DCT32_avx512_LOOP 17, 18, 4, 51005
+ DCT32_avx512_PASS_1_O 1 * 64 + 2 * 8, 17, 181006
+1007
+ lea r0, [r0 + 4 * r2]1008
+ DCT32_avx512_LOOP 19, 20, 6, 71009
+ DCT32_avx512_PASS_1_O 1 * 64 + 3 * 8, 19, 201010
+1011
+ lea r0, [r0 + 4 * r2]1012
+ DCT32_avx512_LOOP 21, 22, 8, 91013
+ DCT32_avx512_PASS_1_O 1 * 64 + 4 * 8, 21, 221014
+1015
+ lea r0, [r0 + 4 * r2]1016
+ DCT32_avx512_LOOP 23, 24, 10, 111017
+ DCT32_avx512_PASS_1_O 1 * 64 + 5 * 8, 23, 241018
+1019
+ lea r0, [r0 + 4 * r2]1020
+ DCT32_avx512_LOOP 25, 26, 12, 131021
+ DCT32_avx512_PASS_1_O 1 * 64 + 6 * 8, 25, 261022
+1023
+ lea r0, [r0 + 4 * r2]1024
+ DCT32_avx512_LOOP 27, 28, 14, 151025
+ DCT32_avx512_PASS_1_O 1 * 64 + 7 * 8, 27, 281026
+%endmacro1027
+1028
+%macro DCT32_avx512_PASS_1_ROW_O_1_7 11029
+ vbroadcasti32x8 m9, [r7 + %1 * 32]1030
+1031
+ DCT32_avx512_PASS_1_O %1 * 64 + 0 * 8, 13, 141032
+ DCT32_avx512_PASS_1_O %1 * 64 + 1 * 8, 15, 161033
+ DCT32_avx512_PASS_1_O %1 * 64 + 2 * 8, 17, 181034
+ DCT32_avx512_PASS_1_O %1 * 64 + 3 * 8, 19, 201035
+ DCT32_avx512_PASS_1_O %1 * 64 + 4 * 8, 21, 221036
+ DCT32_avx512_PASS_1_O %1 * 64 + 5 * 8, 23, 241037
+ DCT32_avx512_PASS_1_O %1 * 64 + 6 * 8, 25, 261038
+ DCT32_avx512_PASS_1_O %1 * 64 + 7 * 8, 27, 281039
+%endmacro1040
+1041
+%macro DCT32_avx512_LOOP_EO 41042
+ mova m4, [rsp + 32 * mmsize + %3 * 64]1043
+ vpermw m4, m8, m41044
+ vextracti64x4 ym5, m4, 11045
+1046
+ mova m6, [rsp + 32 * mmsize + %4 * 64]1047
+ vpermw m6, m8, m61048
+ vextracti64x4 ym7, m6, 11049
+1050
+ vinserti64x4 m4, m4, ym6, 11051
+ vinserti64x4 m5, m5, ym7, 11052
+1053
+ psubw m%1, m4, m5 ; EO1054
+ paddw m%2, m4, m5 ; EE1055
+%endmacro1056
+1057
+%macro DCT32_avx512_PASS_1_ROW_EO 21058
+ pmaddwd m29, m%2, m121059
+ vpsrldq m30, m29, 81060
+ vpaddd m30, m291061
+ vpsrldq m29, m30, 41062
+ vpaddd m29, m301063
+1064
+ vpaddd m29, m01065
+ vpsrad m29, DCT_SHIFT1066
+ vpackssdw m29, m291067
+1068
+ vpermw m29, m11, m291069
+ movq [r5 + %1], xm291070
+%endmacro1071
+1072
+%macro DCT32_avx512_PASS_1_ROW_EO_0 01073
+1074
+ mova m8, [dct32_shuf2_AVX512]1075
+ vbroadcasti32x4 m12, [r7 + 2 * 32]1076
+1077
+ DCT32_avx512_LOOP_EO 13, 14, 0, 11078
+ DCT32_avx512_PASS_1_ROW_EO 2 * 64 + 0 * 8, 131079
+1080
+ lea r9, [r9 + 4 * r2]1081
+ DCT32_avx512_LOOP_EO 15, 16, 2, 31082
+ DCT32_avx512_PASS_1_ROW_EO 2 * 64 + 1 * 8, 151083
+1084
+ lea r9, [r9 + 4 * r2]1085
+ DCT32_avx512_LOOP_EO 17, 18, 4, 51086
+ DCT32_avx512_PASS_1_ROW_EO 2 * 64 + 2 * 8, 171087
+1088
+ lea r9, [r9 + 4 * r2]1089
+ DCT32_avx512_LOOP_EO 19, 20, 6, 71090
+ DCT32_avx512_PASS_1_ROW_EO 2 * 64 + 3 * 8, 191091
+1092
+ lea r9, [r9 + 4 * r2]1093
+ DCT32_avx512_LOOP_EO 21, 22, 8, 91094
+ DCT32_avx512_PASS_1_ROW_EO 2 * 64 + 4 * 8, 211095
+1096
+ lea r9, [r9 + 4 * r2]1097
+ DCT32_avx512_LOOP_EO 23, 24, 10, 111098
+ DCT32_avx512_PASS_1_ROW_EO 2 * 64 + 5 * 8, 231099
+1100
+ lea r9, [r9 + 4 * r2]1101
+ DCT32_avx512_LOOP_EO 25, 26, 12, 131102
+ DCT32_avx512_PASS_1_ROW_EO 2 * 64 + 6 * 8, 251103
+1104
+ lea r9, [r9 + 4 * r2]1105
+ DCT32_avx512_LOOP_EO 27, 28, 14, 151106
+ DCT32_avx512_PASS_1_ROW_EO 2 * 64 + 7 * 8, 271107
+1108
+%endmacro1109
+1110
+%macro DCT32_avx512_PASS_1_ROW_EO_1_7 11111
+1112
+ vbroadcasti32x4 m12, [r7 + %1 * 32]1113
+1114
+ DCT32_avx512_PASS_1_ROW_EO %1 * 64 + 0 * 8, 131115
+ DCT32_avx512_PASS_1_ROW_EO %1 * 64 + 1 * 8, 151116
+ DCT32_avx512_PASS_1_ROW_EO %1 * 64 + 2 * 8, 171117
+ DCT32_avx512_PASS_1_ROW_EO %1 * 64 + 3 * 8, 191118
+ DCT32_avx512_PASS_1_ROW_EO %1 * 64 + 4 * 8, 211119
+ DCT32_avx512_PASS_1_ROW_EO %1 * 64 + 5 * 8, 231120
+ DCT32_avx512_PASS_1_ROW_EO %1 * 64 + 6 * 8, 251121
+ DCT32_avx512_PASS_1_ROW_EO %1 * 64 + 7 * 8, 271122
+1123
+%endmacro1124
+1125
+%macro DCT32_avx512_LOOP_EEO 01126
+ vpunpcklqdq m2, m14, m161127
+ vpunpckhqdq m14, m161128
+ vpshufb m14, m311129
+1130
+ vpaddw m16, m2, m14 ; EEE1131
+ vpsubw m2, m14 ; EE01132
+1133
+ vpunpcklqdq m3, m18, m201134
+ vpunpckhqdq m18, m201135
+ vpshufb m18, m311136
+1137
+ vpaddw m20, m3, m18 ; EEE1138
+ vpsubw m3, m18 ; EE01139
+1140
+ vpunpcklqdq m4, m22, m241141
+ vpunpckhqdq m22, m241142
+ vpshufb m22, m311143
+1144
+ vpaddw m24, m4, m22 ; EEE1145
+ vpsubw m4, m22 ; EE01146
+1147
+ vpunpcklqdq m5, m26, m281148
+ vpunpckhqdq m26, m281149
+ vpshufb m26, m311150
+1151
+ vpaddw m28, m5, m26 ; EEE1152
+ vpsubw m5, m26 ; EE01153
+%endmacro1154
+1155
+%macro DCT32_avx512_PASS_1_ROW_EEO 21156
+ pmaddwd m30, m%2, m11157
+ vpsrldq m29, m30, 41158
+ vpaddd m29, m301159
+1160
+ vpaddd m29, m01161
+ vpsrad m29, DCT_SHIFT1162
+ vpackssdw m29, m291163
+1164
+ vpermw m29, m27, m291165
+ movu [r5 + %1], xm291166
+%endmacro1167
+1168
+%macro DCT32_avx512_PASS_1_ROW_EEO_1_4 11169
+1170
+vpbroadcastq m1, [r7 + %1 * 32]1171
+DCT32_avx512_PASS_1_ROW_EEO %1 * 64 + 0 * 16, 21172
+DCT32_avx512_PASS_1_ROW_EEO %1 * 64 + 1 * 16, 31173
+DCT32_avx512_PASS_1_ROW_EEO %1 * 64 + 2 * 16, 41174
+DCT32_avx512_PASS_1_ROW_EEO %1 * 64 + 3 * 16, 51175
+1176
+%endmacro1177
+1178
+%macro DCT32_avx512_PASS_1_ROW_EEEO_1_4 11179
+1180
+vpbroadcastq m1, [r7 + %1 * 32]1181
+DCT32_avx512_PASS_1_ROW_EEO %1 * 64 + 0 * 16, 161182
+DCT32_avx512_PASS_1_ROW_EEO %1 * 64 + 1 * 16, 201183
+DCT32_avx512_PASS_1_ROW_EEO %1 * 64 + 2 * 16, 241184
+DCT32_avx512_PASS_1_ROW_EEO %1 * 64 + 3 * 16, 281185
+1186
+%endmacro1187
+1188
+%macro DCT32_avx512_PASS2_OPT 51189
+ pmaddwd m9, m1, m%11190
+ vpsrldq m10, m9, 81191
+ vpaddd m9, m101192
+1193
+ pmaddwd m10, m1, m%21194
+ vpsrldq m11, m10, 81195
+ vpaddd m10, m111196
+1197
+ pmaddwd m11, m1, m%31198
+ vpsrldq m12, m11, 81199
+ vpaddd m11, m121200
+1201
+ pmaddwd m12, m1, m%41202
+ vpsrldq m13, m12, 81203
+ vpaddd m12, m131204
+1205
+ vpsrldq m13, m9, 41206
+ vpaddd m9, m131207
+ vpsrldq m13, m10, 41208
+ vpaddd m10, m131209
+ vpsrldq m13, m11, 41210
+ vpaddd m11, m131211
+ vpsrldq m13, m12, 41212
+ vpaddd m12, m131213
+1214
+ vpermd m9, m31, m91215
+ vpermd m10, m31, m101216
+ vpermd m11, m31, m111217
+ vpermd m12, m31, m121218
+1219
+ vpandd m9, m271220
+ vpandd m10, m301221
+ vpandd m11, m291222
+ vpandd m12, m281223
+1224
+ vpaddd m9, m101225
+ vpaddd m11, m121226
+ vpaddd m9, m111227
+1228
+ vpsrldq m10, m9, 81229
+ vpaddd m9, m101230
+ vpsrldq m10, m9, 41231
+ vpaddd m9, m101232
+1233
+ vpermd m9, m31, m91234
+ vpaddd m9, m01235
+ vpsrad m9, DCT_SHIFT21236
+ vpackssdw m9, m91237
+ movq [r1 + %5], xm91238
+1239
+%endmacro1240
+1241
+%macro DCT32_avx512_PASS2 51242
+1243
+ mova m9, [r5 + %1]1244
+ mova m10, [r5 + %2]1245
+ mova m11, [r5 + %3]1246
+ mova m12, [r5 + %4]1247
+1248
+ pmaddwd m9, m1, m91249
+ vpsrldq m13, m9, 81250
+ vpaddd m9, m131251
+1252
+ pmaddwd m10, m1, m101253
+ vpsrldq m13, m10, 81254
+ vpaddd m10, m131255
+1256
+ pmaddwd m11, m1, m111257
+ vpsrldq m13, m11, 81258
+ vpaddd m11, m131259
+1260
+ pmaddwd m12, m1, m121261
+ vpsrldq m13, m12, 81262
+ vpaddd m12, m131263
+1264
+ vpsrldq m13, m9, 41265
+ vpaddd m9, m131266
+ vpsrldq m13, m10, 41267
+ vpaddd m10, m131268
+ vpsrldq m13, m11, 41269
+ vpaddd m11, m131270
+ vpsrldq m13, m12, 41271
+ vpaddd m12, m131272
+1273
+ vpermd m9, m31, m91274
+ vpermd m10, m31, m101275
+ vpermd m11, m31, m111276
+ vpermd m12, m31, m121277
+1278
+ vpandd m9, m271279
+ vpandd m10, m301280
+ vpandd m11, m291281
+ vpandd m12, m281282
+1283
+ vpaddd m9, m101284
+ vpaddd m11, m121285
+ vpaddd m9, m111286
+1287
+ vpsrldq m10, m9, 81288
+ vpaddd m9, m101289
+ vpsrldq m10, m9, 41290
+ vpaddd m9, m101291
+1292
+ vpermd m9, m31, m91293
+ vpaddd m9, m01294
+ vpsrad m9, DCT_SHIFT21295
+ vpackssdw m9, m91296
+ movq [r1 + %5], xm91297
+1298
+%endmacro1299
+1300
+%macro DCT32_avx512_PASS2_1_ROW 11301
+1302
+mova m1, [r8 + %1 * 64]1303
+1304
+DCT32_avx512_PASS2_OPT 2, 3, 4, 14, %1 * 64 + 0 * 81305
+DCT32_avx512_PASS2_OPT 15, 16, 17, 18, %1 * 64 + 1 * 81306
+DCT32_avx512_PASS2_OPT 19, 20, 21, 22, %1 * 64 + 2 * 81307
+DCT32_avx512_PASS2_OPT 23, 24, 25, 26, %1 * 64 + 3 * 81308
+DCT32_avx512_PASS2_OPT 5, 6, 7, 8, %1 * 64 + 4 * 81309
+1310
+DCT32_avx512_PASS2 20 * 64, 21 * 64, 22 * 64, 23 * 64, %1 * 64 + 5 * 81311
+DCT32_avx512_PASS2 24 * 64, 25 * 64, 26 * 64, 27 * 64, %1 * 64 + 6 * 81312
+DCT32_avx512_PASS2 28 * 64, 29 * 64, 30 * 64, 31 * 64, %1 * 64 + 7 * 81313
+1314
+%endmacro1315
+1316
+INIT_ZMM avx5121317
+cglobal dct32, 3, 10, 32, 0-(32*mmsize + 16*mmsize)1318
+1319
+%if BIT_DEPTH == 121320
+ %define DCT_SHIFT 81321
+ vpbroadcastq m0, [pd_128]1322
+%elif BIT_DEPTH == 101323
+ %define DCT_SHIFT 61324
+ vpbroadcastq m0, [pd_32]1325
+%elif BIT_DEPTH == 81326
+ %define DCT_SHIFT 41327
+ vpbroadcastq m0, [pd_8]1328
+%else1329
+ %error Unsupported BIT_DEPTH!1330
+%endif1331
+%define DCT_SHIFT2 111332
+1333
+ add r2d, r2d1334
+ lea r7, [tab_dct32_1]1335
+ lea r8, [tab_dct32]1336
+ lea r3, [r2 * 3]1337
+ mov r5, rsp1338
+ mov r9, 2048 ; 32 * mmsize1339
+ add r9, rsp1340
+1341
+ mova m31, [dct32_shuf1_AVX512]1342
+1343
+ ; PASSS 11344
+1345
+ vbroadcasti32x8 m30, [dct8_shuf9_AVX512]1346
+ mova m8, [dct32_shuf_AVX512]1347
+1348
+ DCT32_avx512_PASS_1_ROW_O1349
+ DCT32_avx512_PASS_1_ROW_O_1_7 31350
+ DCT32_avx512_PASS_1_ROW_O_1_7 51351
+ DCT32_avx512_PASS_1_ROW_O_1_7 71352
+ DCT32_avx512_PASS_1_ROW_O_1_7 91353
+ DCT32_avx512_PASS_1_ROW_O_1_7 111354
+ DCT32_avx512_PASS_1_ROW_O_1_7 131355
+ DCT32_avx512_PASS_1_ROW_O_1_7 151356
+ DCT32_avx512_PASS_1_ROW_O_1_7 171357
+ DCT32_avx512_PASS_1_ROW_O_1_7 191358
+ DCT32_avx512_PASS_1_ROW_O_1_7 201359
+ DCT32_avx512_PASS_1_ROW_O_1_7 211360
+ DCT32_avx512_PASS_1_ROW_O_1_7 231361
+ DCT32_avx512_PASS_1_ROW_O_1_7 251362
+ DCT32_avx512_PASS_1_ROW_O_1_7 271363
+ DCT32_avx512_PASS_1_ROW_O_1_7 291364
+ DCT32_avx512_PASS_1_ROW_O_1_7 311365
+1366
+ vbroadcasti32x8 m11, [dct8_shuf9_AVX512]1367
+1368
+ DCT32_avx512_PASS_1_ROW_EO_01369
+ DCT32_avx512_PASS_1_ROW_EO_1_7 61370
+ DCT32_avx512_PASS_1_ROW_EO_1_7 101371
+ DCT32_avx512_PASS_1_ROW_EO_1_7 141372
+ DCT32_avx512_PASS_1_ROW_EO_1_7 181373
+ DCT32_avx512_PASS_1_ROW_EO_1_7 221374
+ DCT32_avx512_PASS_1_ROW_EO_1_7 261375
+ DCT32_avx512_PASS_1_ROW_EO_1_7 301376
+1377
+ vbroadcasti32x4 m31, [dct8_shuf]1378
+ vbroadcasti32x8 m27, [dct32_shuf3_AVX512]1379
+1380
+ DCT32_avx512_LOOP_EEO1381
+ DCT32_avx512_PASS_1_ROW_EEO_1_4 41382
+ DCT32_avx512_PASS_1_ROW_EEO_1_4 121383
+ DCT32_avx512_PASS_1_ROW_EEO_1_4 201384
+ DCT32_avx512_PASS_1_ROW_EEO_1_4 281385
+1386
+ DCT32_avx512_PASS_1_ROW_EEEO_1_4 01387
+ DCT32_avx512_PASS_1_ROW_EEEO_1_4 161388
+ DCT32_avx512_PASS_1_ROW_EEEO_1_4 81389
+ DCT32_avx512_PASS_1_ROW_EEEO_1_4 241390
+1391
+ ; PASS 21392
+1393
+ vpbroadcastq m0, [pd_1024]1394
+ vbroadcasti32x8 m31, [dct32_shuf4_AVX512]1395
+ movu m30, [dct32_shuf5_AVX512]1396
+ movu m29, [dct32_shuf6_AVX512]1397
+ movu m28, [dct32_shuf7_AVX512]1398
+ movu m27, [dct32_shuf8_AVX512]1399
+1400
+ ;Load the source coefficents into free registers and reuse them for all rows1401
+1402
+ mova m2, [r5 + 0 * 64]1403
+ mova m3, [r5 + 1 * 64]1404
+ mova m4, [r5 + 2 * 64]1405
+ mova m14, [r5 + 3 * 64]1406
+ mova m15, [r5 + 4 * 64]1407
+ mova m16, [r5 + 5 * 64]1408
+ mova m17, [r5 + 6 * 64]1409
+ mova m18, [r5 + 7 * 64]1410
+ mova m19, [r5 + 8 * 64]1411
+ mova m20, [r5 + 9 * 64]1412
+ mova m21, [r5 + 10 * 64]1413
+ mova m22, [r5 + 11 * 64]1414
+ mova m23, [r5 + 12 * 64]1415
+ mova m24, [r5 + 13 * 64]1416
+ mova m25, [r5 + 14 * 64]1417
+ mova m26, [r5 + 15 * 64]1418
+ mova m5, [r5 + 16 * 64]1419
+ mova m6, [r5 + 17 * 64]1420
+ mova m7, [r5 + 18 * 64]1421
+ mova m8, [r5 + 19 * 64]1422
+1423
+ DCT32_avx512_PASS2_1_ROW 01424
+ DCT32_avx512_PASS2_1_ROW 11425
+ DCT32_avx512_PASS2_1_ROW 21426
+ DCT32_avx512_PASS2_1_ROW 31427
+ DCT32_avx512_PASS2_1_ROW 41428
+ DCT32_avx512_PASS2_1_ROW 51429
+ DCT32_avx512_PASS2_1_ROW 61430
+ DCT32_avx512_PASS2_1_ROW 71431
+ DCT32_avx512_PASS2_1_ROW 81432
+ DCT32_avx512_PASS2_1_ROW 91433
+ DCT32_avx512_PASS2_1_ROW 101434
+ DCT32_avx512_PASS2_1_ROW 111435
+ DCT32_avx512_PASS2_1_ROW 121436
+ DCT32_avx512_PASS2_1_ROW 131437
+ DCT32_avx512_PASS2_1_ROW 141438
+ DCT32_avx512_PASS2_1_ROW 151439
+ DCT32_avx512_PASS2_1_ROW 161440
+ DCT32_avx512_PASS2_1_ROW 171441
+ DCT32_avx512_PASS2_1_ROW 181442
+ DCT32_avx512_PASS2_1_ROW 191443
+ DCT32_avx512_PASS2_1_ROW 201444
+ DCT32_avx512_PASS2_1_ROW 211445
+ DCT32_avx512_PASS2_1_ROW 221446
+ DCT32_avx512_PASS2_1_ROW 231447
+ DCT32_avx512_PASS2_1_ROW 241448
+ DCT32_avx512_PASS2_1_ROW 251449
+ DCT32_avx512_PASS2_1_ROW 261450
+ DCT32_avx512_PASS2_1_ROW 271451
+ DCT32_avx512_PASS2_1_ROW 281452
+ DCT32_avx512_PASS2_1_ROW 291453
+ DCT32_avx512_PASS2_1_ROW 301454
+ DCT32_avx512_PASS2_1_ROW 311455
+1456
+ RET1457
+1458
%macro IDCT8_PASS_1 11459
vpbroadcastd m7, [r5 + %1]1460
vpbroadcastd m10, [r5 + %1 + 4]1461
1462
mova [r1 + r3], xm31463
RET1464
1465
+1466
+%macro IDCT8_AVX512_PASS_1 01467
+ pmaddwd m5, m29, m171468
+ pmaddwd m6, m25, m181469
+ paddd m5, m61470
+1471
+ pmaddwd m6, m30, m211472
+ pmaddwd m3, m26, m221473
+ paddd m6, m31474
+1475
+ paddd m3, m5, m61476
+ paddd m3, m111477
+ psrad m3, IDCT_SHIFT11478
+1479
+ psubd m5, m61480
+ paddd m5, m111481
+ psrad m5, IDCT_SHIFT11482
+1483
+ pmaddwd m6, m29, m191484
+ pmaddwd m8, m25, m201485
+ paddd m6, m81486
+1487
+ pmaddwd m8, m30, m231488
+ pmaddwd m9, m26, m241489
+ paddd m8, m91490
+1491
+ paddd m9, m6, m81492
+ paddd m9, m111493
+ psrad m9, IDCT_SHIFT11494
+1495
+ psubd m6, m81496
+ paddd m6, m111497
+ psrad m6, IDCT_SHIFT11498
+1499
+ packssdw m3, m91500
+ vpermq m3, m3, 0xD81501
+1502
+ packssdw m6, m51503
+ vpermq m6, m6, 0xD81504
+%endmacro1505
+1506
+1507
+%macro IDCT8_AVX512_PASS_2 01508
+ mov r7d, 0xAAAA1509
+ kmovd k1, r7d1510
+ punpcklqdq m2, m3, m131511
+ punpckhqdq m0, m3, m131512
+1513
+ pmaddwd m3, m2, [r5]1514
+ pmaddwd m5, m2, [r5 + 1 * mmsize]1515
+ pmaddwd m6, m2, [r5 + 2 * mmsize]1516
+ pmaddwd m7, m2, [r5 + 3 * mmsize]1517
+1518
+ vpsrldq m14, m3, 41519
+ paddd m3, m141520
+ vpslldq m16, m5, 41521
+ paddd m5, m161522
+ vmovdqu32 m3 {k1}, m51523
+1524
+ vpsrldq m14, m6, 41525
+ paddd m6, m141526
+ vpslldq m16, m7, 41527
+ paddd m7, m161528
+ vmovdqu32 m6 {k1}, m71529
+1530
+ punpcklqdq m7, m3, m61531
+ punpckhqdq m3, m61532
+1533
+ pmaddwd m5, m0, [r6]1534
+ pmaddwd m6, m0, [r6 + 1 * mmsize]1535
+ pmaddwd m8, m0, [r6 + 2 * mmsize]1536
+ pmaddwd m9, m0, [r6 + 3 * mmsize]1537
+1538
+ vpsrldq m14, m5, 41539
+ paddd m5, m141540
+ vpslldq m16, m6, 41541
+ paddd m6, m161542
+ vmovdqu32 m5 {k1}, m61543
+1544
+ vpsrldq m14, m8, 41545
+ paddd m8, m141546
+ vpslldq m16, m9, 41547
+ paddd m9, m161548
+ vmovdqu32 m8 {k1}, m91549
+1550
+ punpcklqdq m6, m5, m81551
+ punpckhqdq m5, m81552
+1553
+ paddd m8, m7, m61554
+ paddd m8, m121555
+ psrad m8, IDCT_SHIFT21556
+1557
+ psubd m7, m61558
+ paddd m7, m121559
+ psrad m7, IDCT_SHIFT21560
+1561
+ pshufb m7, [idct8_avx512_shuf3]1562
+ packssdw m8, m71563
+1564
+ paddd m9, m3, m51565
+ paddd m9, m121566
+ psrad m9, IDCT_SHIFT21567
+1568
+ psubd m3, m51569
+ paddd m3, m121570
+ psrad m3, IDCT_SHIFT21571
+1572
+ pshufb m3, [idct8_avx512_shuf3]1573
+ packssdw m9, m31574
+%endmacro1575
+1576
+1577
+%if ARCH_X86_641578
+INIT_ZMM avx5121579
+cglobal idct8, 3, 8, 311580
+%if BIT_DEPTH == 121581
+ %define IDCT_SHIFT2 81582
+ vpbroadcastd m12, [pd_128]1583
+%elif BIT_DEPTH == 101584
+ %define IDCT_SHIFT2 101585
+ vpbroadcastd m12, [pd_512]1586
+%elif BIT_DEPTH == 81587
+ %define IDCT_SHIFT2 121588
+ vpbroadcastd m12, [pd_2048]1589
+%else1590
+ %error Unsupported BIT_DEPTH!1591
+%endif1592
+%define IDCT_SHIFT1 71593
+1594
+ vpbroadcastd m11, [pd_64]1595
+1596
+ lea r4, [avx512_idct8_3]1597
+ lea r5, [avx2_idct8_1]1598
+ lea r6, [avx2_idct8_2]1599
+ movu m16, [idct16_shuff2]1600
+ movu m17, [idct16_shuff3]1601
+1602
+ ;pass11603
+ mova ym1, [r0 + 0 * 32]1604
+ mova ym0, [r0 + 1 * 32]1605
+ mova ym25, ym161606
+ mova ym26, ym171607
+ vpermi2w ym25, ym1, ym01608
+ vpermi2w ym26, ym1, ym01609
+1610
+ mova ym1, [r0 + 2 * 32]1611
+ mova ym0, [r0 + 3 * 32]1612
+ mova ym27, ym161613
+ mova ym28, ym171614
+ vpermi2w ym27, ym1, ym01615
+ vpermi2w ym28, ym1, ym01616
+ 1617
+ vperm2i128 ym29, ym25, ym26, 0x201618
+ vperm2i128 ym30, ym25, ym26, 0x311619
+ vperm2i128 ym25, ym27, ym28, 0x201620
+ vperm2i128 ym26, ym27, ym28, 0x311621
+1622
+ vinserti64x4 m29, m29, ym29, 11623
+ vinserti64x4 m25, m25, ym25, 11624
+ vinserti64x4 m30, m30, ym30, 11625
+ vinserti64x4 m26, m26, ym26, 11626
+1627
+ movu m17, [r4]1628
+ movu m18, [r4 + 1 * mmsize]1629
+ movu m19, [r4 + 2 * mmsize]1630
+ movu m20, [r4 + 3 * mmsize]1631
+ movu m21, [r4 + 4 * mmsize]1632
+ movu m22, [r4 + 5 * mmsize]1633
+ movu m23, [r4 + 6 * mmsize]1634
+ movu m24, [r4 + 7 * mmsize]1635
+1636
+ IDCT8_AVX512_PASS_11637
+1638
+ vextracti64x4 ym13, m3, 11639
+ vextracti64x4 ym14, m6, 11640
+ vinserti64x4 m3, m3, ym14, 11641
+ vinserti64x4 m13, m13, ym6, 11642
+1643
+ ;pass21644
+ add r2d, r2d1645
+ lea r3, [r2 * 3]1646
+ lea r5, [avx512_idct8_1]1647
+ lea r6, [avx512_idct8_2]1648
+1649
+ IDCT8_AVX512_PASS_21650
+1651
+ vextracti128 xm3, ym8, 11652
+ mova [r1], xm81653
+ mova [r1 + r2], xm31654
+ vextracti128 xm3, ym9, 11655
+ mova [r1 + r2 * 2], xm91656
+ mova [r1 + r3], xm31657
+1658
+ lea r1, [r1 + r2 * 4]1659
+1660
+ vextracti64x4 ym10, m8, 11661
+ vextracti64x4 ym11, m9, 11662
+1663
+ vextracti128 xm3, ym10, 11664
+ mova [r1], xm101665
+ mova [r1 + r2], xm31666
+ vextracti128 xm3, ym11, 11667
+ mova [r1 + r2 * 2], xm111668
+ mova [r1 + r3], xm31669
+ RET1670
+%endif1671
+1672
%macro IDCT_PASS1 21673
vbroadcasti128 m5, [tab_idct16_2 + %1 * 16]1674
1675
1676
jnz .pass21677
RET1678
1679
+1680
+%macro IDCT16_AVX512_PASS1 31681
+ movu m5, [tab_AVX512_idct16_2 + %1 * 64]1682
+ pmaddwd m9, m4, m51683
+ pmaddwd m10, m6, m51684
+1685
+ vpsrldq m16, m9, 41686
+ paddd m9, m161687
+ vpslldq m17, m10, 41688
+ paddd m10, m171689
+ vmovdqu32 m9 {k1}, m101690
+1691
+ pmaddwd m10, m7, m51692
+ pmaddwd m11, m8, m51693
+1694
+ vpsrldq m16, m10, 41695
+ paddd m10, m161696
+ vpslldq m17, m11, 41697
+ paddd m11, m171698
+ vmovdqu32 m10 {k1}, m111699
+1700
+ vpsrldq m16, m9, 81701
+ paddd m9, m161702
+ vpslldq m17, m10, 81703
+ paddd m10, m171704
+ vmovdqu32 m9 {k2}, m101705
+1706
+ mova m5, [tab_AVX512_idct16_1 + %1 * 64]1707
+ pmaddwd m10, m28, m51708
+ pmaddwd m11, m29, m51709
+1710
+ vpsrldq m16, m10, 41711
+ paddd m10, m161712
+ vpslldq m17, m11, 41713
+ paddd m11, m171714
+ vmovdqu32 m10 {k1}, m111715
+1716
+ pmaddwd m11, m30, m51717
+ pmaddwd m12, m31, m51718
+1719
+ vpsrldq m16, m11, 41720
+ paddd m11, m161721
+ vpslldq m17, m12, 41722
+ paddd m12, m171723
+ vmovdqu32 m11 {k1}, m121724
+1725
+ vpsrldq m16, m10, 81726
+ paddd m10, m161727
+ vpslldq m17, m11, 81728
+ paddd m11, m171729
+ vmovdqu32 m10 {k2}, m111730
+1731
+ paddd m11, m9, m101732
+ paddd m11, m141733
+ psrad m11, IDCT_SHIFT11734
+1735
+ psubd m9, m101736
+ paddd m9, m141737
+ psrad m9, IDCT_SHIFT11738
+1739
+ mova m5, [tab_AVX512_idct16_2 + %1 * 64 + 64]1740
+ pmaddwd m10, m4, m51741
+ pmaddwd m12, m6, m51742
+1743
+1744
+ vpsrldq m16, m10, 41745
+ paddd m10, m161746
+ vpslldq m17, m12, 41747
+ paddd m12, m171748
+ vmovdqu32 m10 {k1}, m121749
+1750
+ pmaddwd m12, m7, m51751
+ pmaddwd m13, m8, m51752
+1753
+1754
+ vpsrldq m16, m12, 41755
+ paddd m12, m161756
+ vpslldq m17, m13, 41757
+ paddd m13, m171758
+ vmovdqu32 m12 {k1}, m131759
+1760
+1761
+ vpsrldq m16, m10, 81762
+ paddd m10, m161763
+ vpslldq m17, m12, 81764
+ paddd m12, m171765
+ vmovdqu32 m10 {k2}, m121766
+1767
+1768
+1769
+ mova m5, [tab_AVX512_idct16_1 + %1 * 64 + 64] 1770
+ pmaddwd m12, m28, m51771
+ pmaddwd m13, m29, m51772
+1773
+1774
+ vpsrldq m16, m12, 41775
+ paddd m12, m161776
+ vpslldq m17, m13, 41777
+ paddd m13, m171778
+ vmovdqu32 m12 {k1}, m131779
+1780
+ pmaddwd m13, m30, m51781
+ pmaddwd m5, m311782
+1783
+1784
+ vpsrldq m16, m13, 41785
+ paddd m13, m161786
+ vpslldq m17, m5, 41787
+ paddd m5, m171788
+ vmovdqu32 m13 {k1}, m51789
+1790
+1791
+ vpsrldq m16, m12, 81792
+ paddd m12, m161793
+ vpslldq m17, m13, 81794
+ paddd m13, m171795
+ vmovdqu32 m12 {k2}, m131796
+1797
+1798
+ paddd m5, m10, m121799
+ paddd m5, m141800
+ psrad m5, IDCT_SHIFT11801
+1802
+ psubd m10, m121803
+ paddd m10, m141804
+ psrad m10, IDCT_SHIFT11805
+1806
+ packssdw m11, m51807
+ packssdw m9, m101808
+1809
+ mova m10, [idct16_AVX512_shuff]1810
+ mova m5, [idct16_AVX512_shuff1]1811
+1812
+ vpermd m%2, m10, m111813
+ vpermd m%3, m5, m91814
+%endmacro1815
+1816
+%macro IDCT16_AVX512_PASS2 21817
+ vpermq m0, m%1, 0xD81818
+1819
+ pmaddwd m1, m0, m71820
+ pmaddwd m2, m0, m81821
+1822
+1823
+ vpsrldq m14, m1, 41824
+ paddd m1, m141825
+ vpslldq m31, m2, 41826
+ paddd m2, m311827
+ vmovdqu32 m1 {k1}, m21828
+1829
+ pmaddwd m2, m0, m91830
+ pmaddwd m3, m0, m101831
+1832
+1833
+ vpsrldq m14, m2, 41834
+ paddd m2, m141835
+ vpslldq m31, m3, 41836
+ paddd m3, m311837
+ vmovdqu32 m2 {k1}, m31838
+1839
+1840
+ vpsrldq m14, m1, 81841
+ paddd m1, m141842
+ vpslldq m31, m2, 81843
+ paddd m2, m311844
+ vmovdqu32 m1 {k2}, m21845
+1846
+ pmaddwd m2, m0, m111847
+ pmaddwd m3, m0, m121848
+1849
+1850
+ vpsrldq m14, m2, 41851
+ paddd m2, m141852
+ vpslldq m31, m3, 41853
+ paddd m3, m311854
+ vmovdqu32 m2 {k1}, m31855
+1856
+ vbroadcasti64x2 m14, [r5 + 112]1857
+ pmaddwd m3, m0, m131858
+ pmaddwd m4, m0, m141859
+1860
+1861
+ vpsrldq m14, m3, 41862
+ paddd m3, m141863
+ vpslldq m31, m4, 41864
+ paddd m4, m311865
+ vmovdqu32 m3 {k1}, m41866
+1867
+1868
+ vpsrldq m14, m2, 81869
+ paddd m2, m141870
+ vpslldq m31, m3, 81871
+ paddd m3, m311872
+ vmovdqu32 m2 {k2}, m31873
+1874
+ vpermq m0, m%2, 0xD81875
+ pmaddwd m3, m0, m161876
+ pmaddwd m4, m0, m171877
+1878
+1879
+ vpsrldq m14, m3, 41880
+ paddd m3, m141881
+ vpslldq m31, m4, 41882
+ paddd m4, m311883
+ vmovdqu32 m3 {k1}, m41884
+1885
+ pmaddwd m4, m0, m191886
+ pmaddwd m5, m0, m231887
+1888
+1889
+ vpsrldq m14, m4, 41890
+ paddd m4, m141891
+ vpslldq m31, m5, 41892
+ paddd m5, m311893
+ vmovdqu32 m4 {k1}, m51894
+1895
+1896
+ vpsrldq m14, m3, 81897
+ paddd m3, m141898
+ vpslldq m31, m4, 81899
+ paddd m4, m311900
+ vmovdqu32 m3 {k2}, m41901
+1902
+1903
+ pmaddwd m4, m0, m281904
+ pmaddwd m5, m0, m291905
+1906
+ vpsrldq m14, m4, 41907
+ paddd m4, m141908
+ vpslldq m31, m5, 41909
+ paddd m5, m311910
+ vmovdqu32 m4 {k1}, m51911
+1912
+ pmaddwd m6, m0, m301913
+ vbroadcasti64x2 m31, [r6 + 112]1914
+ pmaddwd m0, m311915
+1916
+1917
+ vpsrldq m14, m6, 41918
+ paddd m6, m141919
+ vpslldq m31, m0, 41920
+ paddd m0, m311921
+ vmovdqu32 m6 {k1}, m01922
+1923
+1924
+ vpsrldq m14, m4, 81925
+ paddd m4, m141926
+ vpslldq m31, m6, 81927
+ paddd m6, m311928
+ vmovdqu32 m4 {k2}, m61929
+1930
+ paddd m5, m1, m31931
+ paddd m5, m151932
+ psrad m5, IDCT_SHIFT21933
+1934
+ psubd m1, m31935
+ paddd m1, m151936
+ psrad m1, IDCT_SHIFT21937
+1938
+ paddd m6, m2, m41939
+ paddd m6, m151940
+ psrad m6, IDCT_SHIFT21941
+1942
+ psubd m2, m41943
+ paddd m2, m151944
+ psrad m2, IDCT_SHIFT21945
+1946
+ packssdw m5, m61947
+ packssdw m1, m21948
+ pshufb m2, m1, [idct16_AVX512_shuff6]1949
+%endmacro1950
+1951
+1952
+;-------------------------------------------------------1953
+; void idct16(const int16_t* src, int16_t* dst, intptr_t dstStride)1954
+;-------------------------------------------------------1955
+INIT_ZMM avx5121956
+cglobal idct16, 3, 8, 321957
+%if BIT_DEPTH == 121958
+ %define IDCT_SHIFT2 81959
+ vpbroadcastd m15, [pd_128]1960
+%elif BIT_DEPTH == 101961
+ %define IDCT_SHIFT2 101962
+ vpbroadcastd m15, [pd_512]1963
+%elif BIT_DEPTH == 81964
+ %define IDCT_SHIFT2 121965
+ vpbroadcastd m15, [pd_2048]1966
+%else1967
+ %error Unsupported BIT_DEPTH!1968
+%endif1969
+%define IDCT_SHIFT1 71970
+1971
+ vpbroadcastd m14, [pd_64]1972
+1973
+ add r2d, r2d1974
+1975
+ mov r7d, 0xAAAA1976
+ kmovd k1, r7d1977
+ mov r7d, 0xCCCC1978
+ kmovd k2, r7d1979
+ mova ym2, [idct16_shuff2]1980
+ mova ym3, [idct16_shuff3]1981
+ mova ym26, [idct16_shuff4]1982
+ mova ym27, [idct16_shuff5]1983
+1984
+.pass1:1985
+ movu xm0, [r0 + 0 * 32]1986
+ vinserti128 ym0, ym0, [r0 + 8 * 32], 11987
+ movu xm1, [r0 + 2 * 32]1988
+ vinserti128 ym1, ym1, [r0 + 10 * 32], 11989
+1990
+ mova ym9, ym21991
+ mova ym10, ym31992
+ vpermi2w ym9, ym0, ym11993
+ vpermi2w ym10, ym0, ym11994
+1995
+ movu xm0, [r0 + 4 * 32]1996
+ vinserti128 ym0, ym0, [r0 + 12 * 32], 11997
+ movu xm1, [r0 + 6 * 32]1998
+ vinserti128 ym1, ym1, [r0 + 14 * 32], 11999
+2000
+ mova ym11, ym22001
+ mova ym12, ym32002
+ vpermi2w ym11, ym0, ym12003
+ vpermi2w ym12, ym0, ym12004
+2005
+ mova ym4, ym262006
+ mova ym6, ym272007
+ vpermi2d ym4, ym9, ym112008
+ vpermi2d ym6, ym9, ym112009
+2010
+ mova ym7, ym262011
+ mova ym8, ym272012
+ vpermi2d ym7, ym10, ym122013
+ vpermi2d ym8, ym10, ym122014
+2015
+ vpermq ym4, ym4, q31202016
+ vpermq ym6, ym6, q31202017
+ vpermq ym7, ym7, q31202018
+ vpermq ym8, ym8, q31202019
+2020
+ movu xm0, [r0 + 1 * 32]2021
+ vinserti128 ym0, ym0, [r0 + 9 * 32], 12022
+ movu xm1, [r0 + 3 * 32]2023
+ vinserti128 ym1, ym1, [r0 + 11 * 32], 12024
+2025
+ mova ym9, ym22026
+ mova ym10, ym32027
+ vpermi2w ym9, ym0, ym12028
+ vpermi2w ym10, ym0, ym12029
+2030
+ movu xm0, [r0 + 5 * 32]2031
+ vinserti128 ym0, ym0, [r0 + 13 * 32], 12032
+ movu xm1, [r0 + 7 * 32]2033
+ vinserti128 ym1, ym1, [r0 + 15 * 32], 12034
+2035
+ mova ym11, ym22036
+ mova ym12, ym32037
+ vpermi2w ym11, ym0, ym12038
+ vpermi2w ym12, ym0, ym12039
+2040
+ mova ym28, ym262041
+ mova ym29, ym272042
+ vpermi2d ym28, ym9, ym112043
+ vpermi2d ym29, ym9, ym112044
+2045
+ mova ym30, ym262046
+ mova ym31, ym272047
+ vpermi2d ym30, ym10, ym122048
+ vpermi2d ym31, ym10, ym122049
+2050
+ vpermq ym28, ym28, q31202051
+ vpermq ym29, ym29, q31202052
+ vpermq ym30, ym30, q31202053
+ vpermq ym31, ym31, q31202054
+2055
+ vinserti64x4 m4, m4, ym4, 12056
+ vinserti64x4 m6, m6, ym6, 12057
+ vinserti64x4 m7, m7, ym7, 12058
+ vinserti64x4 m8, m8, ym8, 12059
+ vinserti64x4 m28, m28, ym28, 12060
+ vinserti64x4 m29, m29, ym29, 12061
+ vinserti64x4 m30, m30, ym30, 12062
+ vinserti64x4 m31, m31, ym31, 12063
+2064
+ IDCT16_AVX512_PASS1 0, 18, 192065
+ IDCT16_AVX512_PASS1 2, 20, 212066
+2067
+ add r0, 162068
+2069
+ movu xm0, [r0 + 0 * 32]2070
+ vinserti128 ym0, ym0, [r0 + 8 * 32], 12071
+ movu xm1, [r0 + 2 * 32]2072
+ vinserti128 ym1, ym1, [r0 + 10 * 32], 12073
+2074
+ mova ym9, ym22075
+ mova ym10, ym32076
+ vpermi2w ym9, ym0, ym12077
+ vpermi2w ym10, ym0, ym12078
+2079
+ movu xm0, [r0 + 4 * 32]2080
+ vinserti128 ym0, ym0, [r0 + 12 * 32], 12081
+ movu xm1, [r0 + 6 * 32]2082
+ vinserti128 ym1, ym1, [r0 + 14 * 32], 12083
+2084
+ mova ym11, ym22085
+ mova ym12, ym32086
+ vpermi2w ym11, ym0, ym12087
+ vpermi2w ym12, ym0, ym12088
+2089
+ mova ym4, ym262090
+ mova ym6, ym272091
+ vpermi2d ym4, ym9, ym112092
+ vpermi2d ym6, ym9, ym112093
+2094
+ mova ym7, ym262095
+ mova ym8, ym272096
+ vpermi2d ym7, ym10, ym122097
+ vpermi2d ym8, ym10, ym122098
+2099
+ vpermq ym4, ym4, q31202100
+ vpermq ym6, ym6, q31202101
+ vpermq ym7, ym7, q31202102
+ vpermq ym8, ym8, q31202103
+2104
+ movu xm0, [r0 + 1 * 32]2105
+ vinserti128 ym0, ym0, [r0 + 9 * 32], 12106
+ movu xm1, [r0 + 3 * 32]2107
+ vinserti128 ym1, ym1, [r0 + 11 * 32], 12108
+2109
+ mova ym9, ym22110
+ mova ym10, ym32111
+ vpermi2w ym9, ym0, ym12112
+ vpermi2w ym10, ym0, ym12113
+2114
+ movu xm0, [r0 + 5 * 32]2115
+ vinserti128 ym0, ym0, [r0 + 13 * 32], 12116
+ movu xm1, [r0 + 7 * 32]2117
+ vinserti128 ym1, ym1, [r0 + 15 * 32], 12118
+2119
+ mova ym11, ym22120
+ mova ym12, ym32121
+ vpermi2w ym11, ym0, ym12122
+ vpermi2w ym12, ym0, ym12123
+2124
+ mova ym28, ym262125
+ mova ym29, ym272126
+ vpermi2d ym28, ym9, ym112127
+ vpermi2d ym29, ym9, ym112128
+2129
+ mova ym30, ym262130
+ mova ym31, ym272131
+ vpermi2d ym30, ym10, ym122132
+ vpermi2d ym31, ym10, ym122133
+2134
+ vpermq ym28, ym28, q31202135
+ vpermq ym29, ym29, q31202136
+ vpermq ym30, ym30, q31202137
+ vpermq ym31, ym31, q31202138
+2139
+ vinserti64x4 m4, m4, ym4, 12140
+ vinserti64x4 m6, m6, ym6, 12141
+ vinserti64x4 m7, m7, ym7, 12142
+ vinserti64x4 m8, m8, ym8, 12143
+ vinserti64x4 m28, m28, ym28, 12144
+ vinserti64x4 m29, m29, ym29, 12145
+ vinserti64x4 m30, m30, ym30, 12146
+ vinserti64x4 m31, m31, ym31, 12147
+2148
+2149
+ IDCT16_AVX512_PASS1 0, 22, 232150
+ IDCT16_AVX512_PASS1 2, 24, 252151
+2152
+ mova m26, [idct16_AVX512_shuff2]2153
+ mova m27, [idct16_AVX512_shuff3]2154
+ vpermi2q m26, m18, m222155
+ vpermi2q m27, m18, m222156
+ mova m18, [idct16_AVX512_shuff2]2157
+ mova m22, [idct16_AVX512_shuff3]2158
+ vpermi2q m18, m20, m242159
+ vpermi2q m22, m20, m242160
+ mova m20, [idct16_AVX512_shuff4]2161
+ mova m24, [idct16_AVX512_shuff5]2162
+ vpermi2q m20, m21, m252163
+ vpermi2q m24, m21, m252164
+ mova m21, [idct16_AVX512_shuff4]2165
+ mova m25, [idct16_AVX512_shuff5]2166
+ vpermi2q m21, m19, m232167
+ vpermi2q m25, m19, m232168
+2169
+ lea r5, [tab_idct16_2]2170
+ lea r6, [tab_idct16_1]2171
+2172
+ vbroadcasti64x2 m7, [r5]2173
+ vbroadcasti64x2 m8, [r5 + 16]2174
+ vbroadcasti64x2 m9, [r5 + 32]2175
+ vbroadcasti64x2 m10, [r5 + 48]2176
+ vbroadcasti64x2 m11, [r5 + 64]2177
+ vbroadcasti64x2 m12, [r5 + 80]2178
+ vbroadcasti64x2 m13, [r5 + 96]2179
+2180
+ vbroadcasti64x2 m16, [r6]2181
+ vbroadcasti64x2 m17, [r6 + 16]2182
+ vbroadcasti64x2 m19, [r6 + 32]2183
+ vbroadcasti64x2 m23, [r6 + 48]2184
+ vbroadcasti64x2 m28, [r6 + 64]2185
+ vbroadcasti64x2 m29, [r6 + 80]2186
+ vbroadcasti64x2 m30, [r6 + 96]2187
+2188
+2189
+ IDCT16_AVX512_PASS2 26, 272190
+ mova [r1], xm52191
+ mova [r1 + 16], xm22192
+ vextracti128 [r1 + r2], ym5, 12193
+ vextracti128 [r1 + r2 + 16], ym2, 12194
+ vextracti64x4 ym14, m5, 12195
+ vextracti64x4 ym31, m2, 12196
+ lea r1, [r1 + 2 * r2]2197
+ mova [r1], xm142198
+ mova [r1 + 16], xm312199
+ vextracti128 [r1 + r2], ym14, 12200
+ vextracti128 [r1 + r2 + 16], ym31, 12201
+2202
+ IDCT16_AVX512_PASS2 18, 222203
+ lea r1, [r1 + 2 * r2]2204
+ mova [r1], xm52205
+ mova [r1 + 16], xm22206
+ vextracti128 [r1 + r2], ym5, 12207
+ vextracti128 [r1 + r2 + 16], ym2, 12208
+ vextracti64x4 ym14, m5, 12209
+ vextracti64x4 ym31, m2, 12210
+ lea r1, [r1 + 2 * r2]2211
+ mova [r1], xm142212
+ mova [r1 + 16], xm312213
+ vextracti128 [r1 + r2], ym14, 12214
+ vextracti128 [r1 + r2 + 16], ym31, 12215
+2216
+ IDCT16_AVX512_PASS2 20, 242217
+ lea r1, [r1 + 2 * r2]2218
+ mova [r1], xm52219
+ mova [r1 + 16], xm22220
+ vextracti128 [r1 + r2], ym5, 12221
+ vextracti128 [r1 + r2 + 16], ym2, 12222
+ vextracti64x4 ym14, m5, 12223
+ vextracti64x4 ym31, m2, 12224
+ lea r1, [r1 + 2 * r2]2225
+ mova [r1], xm142226
+ mova [r1 + 16], xm312227
+ vextracti128 [r1 + r2], ym14, 12228
+ vextracti128 [r1 + r2 + 16], ym31, 12229
+2230
+ IDCT16_AVX512_PASS2 21, 252231
+ lea r1, [r1 + 2 * r2]2232
+ mova [r1], xm52233
+ mova [r1 + 16], xm22234
+ vextracti128 [r1 + r2], ym5, 12235
+ vextracti128 [r1 + r2 + 16], ym2, 12236
+ vextracti64x4 ym14, m5, 12237
+ vextracti64x4 ym31, m2, 12238
+ lea r1, [r1 + 2 * r2]2239
+ mova [r1], xm142240
+ mova [r1 + 16], xm312241
+ vextracti128 [r1 + r2], ym14, 12242
+ vextracti128 [r1 + r2 + 16], ym31, 12243
+ RET2244
+2245
+2246
+2247
%macro IDCT32_PASS1 12248
vbroadcasti128 m3, [tab_idct32_1 + %1 * 32]2249
vbroadcasti128 m13, [tab_idct32_1 + %1 * 32 + 16]2250
2251
jnz .pass22252
RET2253
2254
+2255
+%macro IDCT32_AVX512_PASS1 52256
+ pmaddwd m9, m8, m%42257
+ pmaddwd m10, m7, m%52258
+2259
+ paddd m9, m102260
+ vpsrldq m0, m9, 82261
+ paddd m9, m02262
+ vpsrldq m0, m9, 42263
+ paddd m9, m02264
+2265
+ pmaddwd m10, m4, m%42266
+ pmaddwd m11, m1, m%52267
+2268
+ paddd m10, m112269
+ vpsrldq m0, m10, 82270
+ paddd m10, m02271
+ vpslldq m0, m10, 42272
+ paddd m10, m02273
+2274
+ vmovdqu32 m9 {k3}, m102275
+2276
+ mova m6, [tab_idct32_AVX512_5 + %1 * 64]2277
+ mova m5, [tab_idct32_AVX512_5 + %1 * 64 + 64]2278
+2279
+ pmaddwd m10, m8, m62280
+ pmaddwd m11, m7, m52281
+2282
+ paddd m10, m112283
+ vpslldq m0, m10, 82284
+ paddd m10, m02285
+ vpsrldq m0, m10, 42286
+ paddd m10, m02287
+2288
+ pmaddwd m11, m4, m62289
+ pmaddwd m12, m1, m52290
+2291
+ paddd m11, m122292
+ vpslldq m0, m11, 82293
+ paddd m11, m02294
+ vpslldq m0, m11, 42295
+ paddd m11, m02296
+2297
+ vmovdqu32 m10 {k4}, m112298
+ vmovdqu32 m9 {k2}, m102299
+2300
+ pmaddwd m10, m3, m%22301
+ pmaddwd m11, m14, m%22302
+2303
+ vpsrldq m0, m10, 42304
+ paddd m10, m02305
+ vpslldq m5, m11, 42306
+ paddd m11, m52307
+ vmovdqu32 m10 {k1}, m112308
+2309
+ vpsrldq m0, m10, 82310
+ paddd m10, m02311
+2312
+ pmaddwd m11, m2, m%32313
+ pmaddwd m12, m13, m%32314
+2315
+ vpsrldq m0, m11, 42316
+ paddd m11, m02317
+ vpslldq m5, m12, 42318
+ paddd m12, m52319
+ vmovdqu32 m11 {k1}, m122320
+2321
+ vpsrldq m0, m11, 82322
+ paddd m11, m02323
+2324
+ paddd m12, m10, m112325
+ psubd m10, m112326
+2327
+ punpcklqdq m12, m102328
+ paddd m10, m9, m122329
+ paddd m10, m152330
+ psrad m10, IDCT_SHIFT12331
+2332
+ psubd m12, m92333
+ paddd m12, m152334
+ psrad m12, IDCT_SHIFT12335
+2336
+ packssdw m10, m122337
+ vextracti128 xm12, m10, 12338
+ vextracti64x4 ym5, m10, 12339
+ vextracti128 xm0, ym5, 12340
+2341
+ movd [r3 + %1 * 64], xm102342
+ movd [r3 + 32 + %1 * 64], xm122343
+ pextrd [r4 - %1 * 64], xm10, 12344
+ pextrd [r4+ 32 - %1 * 64], xm12, 12345
+ pextrd [r3 + 16 * 64 + %1 *64], xm10, 32346
+ pextrd [r3 + 16 * 64 + 32 + %1 * 64], xm12, 32347
+ pextrd [r4 + 16 * 64 - %1 * 64], xm10, 22348
+ pextrd [r4 + 16 * 64 + 32 - %1 * 64], xm12, 22349
+2350
+ movd [r3 + (%1 + 1) * 64], xm52351
+ movd [r3 + 32 + (%1 + 1) * 64], xm02352
+ pextrd [r4 - (%1 + 1) * 64], xm5, 12353
+ pextrd [r4+ 32 - (%1 + 1) * 64], xm0, 12354
+ pextrd [r3 + 16 * 64 + (%1 + 1) * 64], xm5, 32355
+ pextrd [r3 + 16 * 64 + 32 + (%1 + 1) * 64], xm0, 32356
+ pextrd [r4 + 16 * 64 - (%1 + 1) * 64], xm5, 22357
+ pextrd [r4 + 16 * 64 + 32 - (%1 + 1) * 64], xm0, 22358
+%endmacro2359
+2360
+%macro IDCT32_AVX512_PASS2 02361
+ pmaddwd m2, m0, m72362
+ pmaddwd m3, m0, m82363
+2364
+ vpsrldq m24, m2, 42365
+ paddd m2, m242366
+ vpslldq m25, m3, 42367
+ paddd m3, m252368
+ vmovdqu32 m2 {k1}, m32369
+2370
+ pmaddwd m3, m0, m92371
+ pmaddwd m4, m0, m102372
+2373
+ vpsrldq m24, m3, 42374
+ paddd m3, m242375
+ vpslldq m25, m4, 42376
+ paddd m4, m252377
+ vmovdqu32 m3 {k1}, m42378
+2379
+ vpsrldq m24, m2, 82380
+ paddd m2, m242381
+ vpslldq m25, m3, 82382
+ paddd m3, m252383
+ vmovdqu32 m2 {k2}, m32384
+2385
+ pmaddwd m3, m0, m112386
+ pmaddwd m4, m0, m122387
+2388
+ vpsrldq m24, m3, 42389
+ paddd m3, m242390
+ vpslldq m25, m4, 42391
+ paddd m4, m252392
+ vmovdqu32 m3 {k1}, m42393
+2394
+ pmaddwd m4, m0, m132395
+ pmaddwd m5, m0, m142396
+2397
+ vpsrldq m24, m4, 42398
+ paddd m4, m242399
+ vpslldq m25, m5, 42400
+ paddd m5, m252401
+ vmovdqu32 m4 {k1}, m52402
+2403
+ vpsrldq m24, m3, 82404
+ paddd m3, m242405
+ vpslldq m25, m4, 82406
+ paddd m4, m252407
+ vmovdqu32 m3 {k2}, m42408
+2409
+ mova m24, [idct16_AVX512_shuff3]2410
+ mova m25, [idct16_AVX512_shuff2]2411
+ vpermi2q m24, m2, m32412
+ vpermi2q m25, m2, m32413
+ paddd m2, m25, m242414
+2415
+ pmaddwd m3, m0, m162416
+ pmaddwd m4, m0, m172417
+2418
+ vpsrldq m24, m3, 42419
+ paddd m3, m242420
+ vpslldq m25, m4, 42421
+ paddd m4, m252422
+ vmovdqu32 m3 {k1}, m42423
+2424
+ pmaddwd m4, m0, m182425
+ pmaddwd m5, m0, m192426
+2427
+ vpsrldq m24, m4, 42428
+ paddd m4, m242429
+ vpslldq m25, m5, 42430
+ paddd m5, m252431
+ vmovdqu32 m4 {k1}, m52432
+2433
+ vpsrldq m24, m3, 82434
+ paddd m3, m242435
+ vpslldq m25, m4, 82436
+ paddd m4, m252437
+ vmovdqu32 m3 {k2}, m42438
+2439
+ pmaddwd m4, m0, m202440
+ pmaddwd m5, m0, m212441
+2442
+ vpsrldq m24, m4, 42443
+ paddd m4, m242444
+ vpslldq m25, m5, 42445
+ paddd m5, m252446
+ vmovdqu32 m4 {k1}, m52447
+2448
+ pmaddwd m5, m0, m222449
+ pmaddwd m0, m232450
+2451
+ vpsrldq m24, m5, 42452
+ paddd m5, m242453
+ vpslldq m25, m0, 42454
+ paddd m0, m252455
+ vmovdqu32 m5 {k1}, m02456
+2457
+ vpsrldq m24, m4, 82458
+ paddd m4, m242459
+ vpslldq m25, m5, 82460
+ paddd m5, m252461
+ vmovdqu32 m4 {k2}, m52462
+2463
+ mova m24, [idct16_AVX512_shuff3]2464
+ mova m25, [idct16_AVX512_shuff2]2465
+ vpermi2q m24, m3, m42466
+ vpermi2q m25, m3, m42467
+ paddd m3, m25, m242468
+2469
+ pmaddwd m4, m1, m262470
+ pmaddwd m0, m1, m272471
+2472
+ vpsrldq m24, m4, 42473
+ paddd m4, m242474
+ vpslldq m25, m0, 42475
+ paddd m0, m252476
+ vmovdqu32 m4 {k1}, m02477
+2478
+ pmaddwd m5, m1, m282479
+ pmaddwd m0, m1, m292480
+2481
+ vpsrldq m24, m5, 42482
+ paddd m5, m242483
+ vpslldq m25, m0, 42484
+ paddd m0, m252485
+ vmovdqu32 m5 {k1}, m02486
+2487
+2488
+ vpsrldq m24, m4, 82489
+ paddd m4, m242490
+ vpslldq m25, m5, 82491
+ paddd m5, m252492
+ vmovdqu32 m4 {k2}, m52493
+2494
+ pmaddwd m5, m1, m302495
+ pmaddwd m0, m1, m312496
+2497
+ vpsrldq m24, m5, 42498
+ paddd m5, m242499
+ vpslldq m25, m0, 42500
+ paddd m0, m252501
+ vmovdqu32 m5 {k1}, m02502
+2503
+ pmaddwd m6, m1, [tab_idct32_AVX512_4 + 6 * mmsize]2504
+ pmaddwd m0, m1, [tab_idct32_AVX512_4 + 7 * mmsize]2505
+2506
+ vpsrldq m24, m6, 42507
+ paddd m6, m242508
+ vpslldq m25, m0, 42509
+ paddd m0, m252510
+ vmovdqu32 m6 {k1}, m02511
+2512
+ vpsrldq m24, m5, 82513
+ paddd m5, m242514
+ vpslldq m25, m6, 82515
+ paddd m6, m252516
+ vmovdqu32 m5 {k2}, m62517
+2518
+ mova m24, [idct16_AVX512_shuff3]2519
+ mova m25, [idct16_AVX512_shuff2]2520
+ vpermi2q m24, m4, m52521
+ vpermi2q m25, m4, m52522
+ paddd m4, m25, m242523
+2524
+ pmaddwd m5, m1, [tab_idct32_AVX512_4 + 8 * mmsize]2525
+ pmaddwd m0, m1, [tab_idct32_AVX512_4 + 9 * mmsize]2526
+2527
+ vpsrldq m24, m5, 42528
+ paddd m5, m242529
+ vpslldq m25, m0, 42530
+ paddd m0, m252531
+ vmovdqu32 m5 {k1}, m02532
+2533
+ pmaddwd m6, m1, [tab_idct32_AVX512_4 + 10 * mmsize]2534
+ pmaddwd m0, m1, [tab_idct32_AVX512_4 + 11 * mmsize]2535
+2536
+ vpsrldq m24, m6, 42537
+ paddd m6, m242538
+ vpslldq m25, m0, 42539
+ paddd m0, m252540
+ vmovdqu32 m6 {k1}, m02541
+2542
+ vpsrldq m24, m5, 82543
+ paddd m5, m242544
+ vpslldq m25, m6, 82545
+ paddd m6, m252546
+ vmovdqu32 m5 {k2}, m62547
+2548
+ pmaddwd m6, m1, [tab_idct32_AVX512_4 + 12 * mmsize]2549
+ pmaddwd m0, m1, [tab_idct32_AVX512_4 + 13 * mmsize]2550
+2551
+ vpsrldq m24, m6, 42552
+ paddd m6, m242553
+ vpslldq m25, m0, 42554
+ paddd m0, m252555
+ vmovdqu32 m6 {k1}, m02556
+2557
+ pmaddwd m0, m1, [tab_idct32_AVX512_4 + 14 * mmsize]2558
+ pmaddwd m1, [tab_idct32_AVX512_4 + 15 * mmsize]2559
+2560
+ vpsrldq m24, m0, 42561
+ paddd m0, m242562
+ vpslldq m25, m1, 42563
+ paddd m1, m252564
+ vmovdqu32 m0 {k1}, m12565
+2566
+ vpsrldq m24, m6, 82567
+ paddd m6, m242568
+ vpslldq m25, m0, 82569
+ paddd m0, m252570
+ vmovdqu32 m6 {k2}, m02571
+2572
+ mova m24, [idct16_AVX512_shuff3]2573
+ mova m25, [idct16_AVX512_shuff2]2574
+ vpermi2q m24, m5, m62575
+ vpermi2q m25, m5, m62576
+ paddd m5, m25, m242577
+2578
+ paddd m6, m2, m42579
+ paddd m6, m152580
+ psrad m6, IDCT_SHIFT22581
+2582
+ psubd m2, m42583
+ paddd m2, m152584
+ psrad m2, IDCT_SHIFT22585
+2586
+ paddd m4, m3, m52587
+ paddd m4, m152588
+ psrad m4, IDCT_SHIFT22589
+2590
+ psubd m3, m52591
+ paddd m3, m152592
+ psrad m3, IDCT_SHIFT22593
+2594
+ packssdw m6, m42595
+ packssdw m2, m32596
+2597
+ vpermq m6, m6, 0xD82598
+ vpermq m2, m2, 0x8D2599
+ pshufb m2, [idct16_AVX512_shuff6]2600
+%endmacro2601
+2602
+;-------------------------------------------------------------------2603
+; void idct32(const int16_t* src, int16_t* dst, intptr_t dstStride)2604
+;-------------------------------------------------------------------2605
+2606
+INIT_ZMM avx5122607
+cglobal idct32, 3, 8, 32, 0-32*642608
+2609
+%define IDCT_SHIFT1 72610
+2611
+ vbroadcasti128 m15, [pd_64]2612
+2613
+ mov r3, rsp2614
+ lea r4, [r3 + 15 * 64]2615
+ mov r5d, 82616
+ mov r7d, 0xAAAA2617
+ kmovd k1, r7d2618
+ mov r7d, 0xCCCC2619
+ kmovd k2, r7d2620
+ mov r7d, 0x22222621
+ kmovd k3, r7d2622
+ mov r7d, 0x88882623
+ kmovd k4, r7d2624
+2625
+2626
+ mova m16, [tab_idct32_AVX512_2 + 0 * 64]2627
+ mova m17, [tab_idct32_AVX512_2 + 1 * 64]2628
+ mova m18, [tab_idct32_AVX512_2 + 2 * 64]2629
+ mova m19, [tab_idct32_AVX512_2 + 3 * 64]2630
+2631
+ mova m20, [tab_idct32_AVX512_3 + 0 * 64]2632
+ mova m21, [tab_idct32_AVX512_3 + 1 * 64]2633
+ mova m22, [tab_idct32_AVX512_3 + 2 * 64]2634
+ mova m23, [tab_idct32_AVX512_3 + 3 * 64]2635
+2636
+ mova m24, [tab_idct32_AVX512_1 + 0 * 64]2637
+ mova m25, [tab_idct32_AVX512_1 + 1 * 64]2638
+ mova m26, [tab_idct32_AVX512_1 + 2 * 64]2639
+ mova m27, [tab_idct32_AVX512_1 + 3 * 64]2640
+ mova m28, [tab_idct32_AVX512_1 + 4 * 64]2641
+ mova m29, [tab_idct32_AVX512_1 + 5 * 64]2642
+ mova m30, [tab_idct32_AVX512_1 + 6 * 64]2643
+ mova m31, [tab_idct32_AVX512_1 + 7 * 64]2644
+2645
+.pass1:2646
+ movq xm0, [r0 + 2 * 64]2647
+ movq xm1, [r0 + 18 * 64]2648
+ punpcklqdq xm0, xm0, xm12649
+ movq xm1, [r0 + 0 * 64]2650
+ movq xm2, [r0 + 16 * 64]2651
+ punpcklqdq xm1, xm1, xm22652
+ vinserti128 ym0, ym0, xm1, 1 ;[2 18 0 16]2653
+2654
+ movq xm1, [r0 + 1 * 64]2655
+ movq xm2, [r0 + 9 * 64]2656
+ punpcklqdq xm1, xm1, xm22657
+ movq xm2, [r0 + 17 * 64]2658
+ movq xm3, [r0 + 25 * 64]2659
+ punpcklqdq xm2, xm2, xm32660
+ vinserti128 ym1, ym1, xm2, 1 ;[1 9 17 25]2661
+2662
+ movq xm2, [r0 + 6 * 64]2663
+ movq xm3, [r0 + 22 * 64]2664
+ punpcklqdq xm2, xm2, xm32665
+ movq xm3, [r0 + 4 * 64]2666
+ movq xm4, [r0 + 20 * 64]2667
+ punpcklqdq xm3, xm3, xm42668
+ vinserti128 ym2, ym2, xm3, 1 ;[6 22 4 20]2669
+2670
+ movq xm3, [r0 + 3 * 64]2671
+ movq xm4, [r0 + 11 * 64]2672
+ punpcklqdq xm3, xm3, xm42673
+ movq xm4, [r0 + 19 * 64]2674
+ movq xm5, [r0 + 27 * 64]2675
+ punpcklqdq xm4, xm4, xm52676
+ vinserti128 ym3, ym3, xm4, 1 ;[3 11 17 25]2677
+2678
+ movq xm4, [r0 + 10 * 64]2679
+ movq xm5, [r0 + 26 * 64]2680
+ punpcklqdq xm4, xm4, xm52681
+ movq xm5, [r0 + 8 * 64]2682
+ movq xm6, [r0 + 24 * 64]2683
+ punpcklqdq xm5, xm5, xm62684
+ vinserti128 ym4, ym4, xm5, 1 ;[10 26 8 24]2685
+2686
+ movq xm5, [r0 + 5 * 64]2687
+ movq xm6, [r0 + 13 * 64]2688
+ punpcklqdq xm5, xm5, xm62689
+ movq xm6, [r0 + 21 * 64]2690
+ movq xm7, [r0 + 29 * 64]2691
+ punpcklqdq xm6, xm6, xm72692
+ vinserti128 ym5, ym5, xm6, 1 ;[5 13 21 9]2693
+2694
+ movq xm6, [r0 + 14 * 64]2695
+ movq xm7, [r0 + 30 * 64]2696
+ punpcklqdq xm6, xm6, xm72697
+ movq xm7, [r0 + 12 * 64]2698
+ movq xm8, [r0 + 28 * 64]2699
+ punpcklqdq xm7, xm7, xm82700
+ vinserti128 ym6, ym6, xm7, 1 ;[14 30 12 28]2701
+2702
+ movq xm7, [r0 + 7 * 64]2703
+ movq xm8, [r0 + 15 * 64]2704
+ punpcklqdq xm7, xm7, xm82705
+ movq xm8, [r0 + 23 * 64]2706
+ movq xm9, [r0 + 31 * 64]2707
+ punpcklqdq xm8, xm8, xm92708
+ vinserti128 ym7, ym7, xm8, 1 ;[7 15 23 31]2709
+2710
+ punpckhwd ym8, ym0, ym2 ;[18 22 16 20]2711
+ punpcklwd ym0, ym2 ;[2 6 0 4]2712
+2713
+ punpckhwd ym2, ym1, ym3 ;[9 11 25 27]2714
+ punpcklwd ym1, ym3 ;[1 3 17 19]2715
+2716
+ punpckhwd ym3, ym4, ym6 ;[26 30 24 28]2717
+ punpcklwd ym4, ym6 ;[10 14 8 12]2718
+2719
+ punpckhwd ym6, ym5, ym7 ;[13 15 29 31]2720
+ punpcklwd ym5, ym7 ;[5 7 21 23]2721
+2722
+ punpckhdq ym7, ym0, ym4 ;[22 62 102 142 23 63 103 143 02 42 82 122 03 43 83 123]2723
+ punpckldq ym0, ym4 ;[20 60 100 140 21 61 101 141 00 40 80 120 01 41 81 121]2724
+2725
+ punpckhdq ym4, ym8, ym3 ;[182 222 262 302 183 223 263 303 162 202 242 282 163 203 243 283]2726
+ punpckldq ym8, ym3 ;[180 220 260 300 181 221 261 301 160 200 240 280 161 201 241 281]2727
+2728
+ punpckhdq ym3, ym1, ym5 ;[12 32 52 72 13 33 53 73 172 192 212 232 173 193 213 233]2729
+ punpckldq ym1, ym5 ;[10 30 50 70 11 31 51 71 170 190 210 230 171 191 211 231]2730
+2731
+ punpckhdq ym5, ym2, ym6 ;[92 112 132 152 93 113 133 153 252 272 292 312 253 273 293 313]2732
+ punpckldq ym2, ym6 ;[90 110 130 150 91 111 131 151 250 270 290 310 251 271 291 311]2733
+2734
+ punpckhqdq ym6, ym0, ym8 ;[21 61 101 141 181 221 261 301 01 41 81 121 161 201 241 281]2735
+ punpcklqdq ym0, ym8 ;[20 60 100 140 180 220 260 300 00 40 80 120 160 200 240 280]2736
+2737
+ punpckhqdq ym8, ym7, ym4 ;[23 63 103 143 183 223 263 303 03 43 83 123 163 203 243 283]2738
+ punpcklqdq ym7, ym4 ;[22 62 102 142 182 222 262 302 02 42 82 122 162 202 242 282]2739
+2740
+ punpckhqdq ym4, ym1, ym2 ;[11 31 51 71 91 111 131 151 171 191 211 231 251 271 291 311]2741
+ punpcklqdq ym1, ym2 ;[10 30 50 70 90 110 130 150 170 190 210 230 250 270 290 310]2742
+2743
+ punpckhqdq ym2, ym3, ym5 ;[13 33 53 73 93 113 133 153 173 193 213 233 253 273 293 313]2744
+ punpcklqdq ym3, ym5 ;[12 32 52 72 92 112 132 152 172 192 212 232 252 272 292 312]2745
+2746
+ vinserti64x4 m7, m7, ym7, 12747
+ vinserti64x4 m8, m8, ym8, 12748
+ movu m13, [idct16_AVX512_shuff2]2749
+ movu m14, [idct16_AVX512_shuff3]2750
+ vpermi2q m13, m7, m82751
+ vpermi2q m14, m7, m82752
+2753
+ vinserti64x4 m1, m1, ym1, 12754
+ vinserti64x4 m4, m4, ym4, 12755
+ movu m7, [idct16_AVX512_shuff3]2756
+ movu m8, [idct16_AVX512_shuff2]2757
+ vpermi2q m7, m1, m42758
+ vpermi2q m8, m1, m42759
+2760
+ vinserti64x4 m3, m3, ym3, 12761
+ vinserti64x4 m2, m2, ym2, 12762
+ movu m1, [idct16_AVX512_shuff3]2763
+ movu m4, [idct16_AVX512_shuff2]2764
+ vpermi2q m1, m3, m22765
+ vpermi2q m4, m3, m22766
+2767
+ vinserti64x4 m0, m0, ym0, 12768
+ vinserti64x4 m6, m6, ym6, 12769
+ movu m2, [idct16_AVX512_shuff2]2770
+ movu m3, [idct16_AVX512_shuff3]2771
+ vpermi2q m2, m0, m62772
+ vpermi2q m3, m0, m62773
+2774
+2775
+ IDCT32_AVX512_PASS1 0, 16, 20, 24, 252776
+ IDCT32_AVX512_PASS1 2, 17, 21, 26, 272777
+ IDCT32_AVX512_PASS1 4, 18, 22, 28, 292778
+ IDCT32_AVX512_PASS1 6, 19, 23, 30, 312779
+2780
+ add r0, 82781
+ add r3, 42782
+ add r4, 42783
+ dec r5d2784
+ jnz .pass12785
+2786
+%if BIT_DEPTH == 122787
+ %define IDCT_SHIFT2 82788
+ vpbroadcastd m15, [pd_128]2789
+%elif BIT_DEPTH == 102790
+ %define IDCT_SHIFT2 102791
+ vpbroadcastd m15, [pd_512]2792
+%elif BIT_DEPTH == 82793
+ %define IDCT_SHIFT2 122794
+ vpbroadcastd m15, [pd_2048]2795
+%else2796
+ %error Unsupported BIT_DEPTH!2797
+%endif2798
+2799
+ mov r3, rsp2800
+ add r2d, r2d2801
+ mov r4d, 162802
+ mov r6d, 0xFFFF00002803
+ kmovd k3, r6d2804
+2805
+ mova m7, [tab_idct32_AVX512_6]2806
+ mova m8, [tab_idct32_AVX512_6 + 1 * mmsize]2807
+ mova m9, [tab_idct32_AVX512_6 + 2 * mmsize]2808
+ mova m10, [tab_idct32_AVX512_6 + 3 * mmsize]2809
+ mova m11, [tab_idct32_AVX512_6 + 4 * mmsize]2810
+ mova m12, [tab_idct32_AVX512_6 + 5 * mmsize]2811
+ mova m13, [tab_idct32_AVX512_6 + 6 * mmsize]2812
+ mova m14, [tab_idct32_AVX512_6 + 7 * mmsize]2813
+ mova m16, [tab_idct32_AVX512_6 + 8 * mmsize]2814
+ mova m17, [tab_idct32_AVX512_6 + 9 * mmsize]2815
+ mova m18, [tab_idct32_AVX512_6 + 10 * mmsize]2816
+ mova m19, [tab_idct32_AVX512_6 + 11 * mmsize]2817
+ mova m20, [tab_idct32_AVX512_6 + 12 * mmsize]2818
+ mova m21, [tab_idct32_AVX512_6 + 13 * mmsize]2819
+ mova m22, [tab_idct32_AVX512_6 + 14 * mmsize]2820
+ mova m23, [tab_idct32_AVX512_6 + 15 * mmsize]2821
+ mova m26, [tab_idct32_AVX512_4]2822
+ mova m27, [tab_idct32_AVX512_4 + 1 * mmsize]2823
+ mova m28, [tab_idct32_AVX512_4 + 2 * mmsize]2824
+ mova m29, [tab_idct32_AVX512_4 + 3 * mmsize]2825
+ mova m30, [tab_idct32_AVX512_4 + 4 * mmsize]2826
+ mova m31, [tab_idct32_AVX512_4 + 5 * mmsize]2827
+2828
+.pass2:2829
+ movu ym0, [r3]2830
+ movu ym1, [r3 + 32]2831
+ vmovdqu16 m0 {k3}, [r3 + 32]2832
+ vmovdqu16 m1 {k3}, [r3 + 64]2833
+2834
+ IDCT32_AVX512_PASS22835
+ movu [r1], ym62836
+ movu [r1 + 32], ym22837
+ vextracti64x4 ym24, m6, 12838
+ vextracti64x4 ym25, m2, 12839
+ add r1, r22840
+ movu [r1 ], ym242841
+ movu [r1 + 32], ym252842
+2843
+ add r1, r22844
+ add r3, 1282845
+ dec r4d2846
+ jnz .pass22847
+ RET2848
+2849
;-------------------------------------------------------2850
; void idct4(const int16_t* src, int16_t* dst, intptr_t dstStride)2851
;-------------------------------------------------------2852
2853
movhps [r1 + 2 * r2], xm02854
movhps [r1 + r3], xm12855
RET2856
+2857
+;static void nonPsyRdoQuant_c(int16_t *m_resiDctCoeff, int64_t *costUncoded, int64_t *totalUncodedCost, int64_t *totalRdCost, uint32_t blkPos)2858
+;{2859
+; const int transformShift = MAX_TR_DYNAMIC_RANGE - X265_DEPTH - log2TrSize; /* Represents scaling through forward transform */2860
+; const int scaleBits = SCALE_BITS - 2 * transformShift;2861
+; const uint32_t trSize = 1 << log2TrSize;2862
+2863
+; for (int y = 0; y < MLS_CG_SIZE; y++)2864
+; {2865
+; for (int x = 0; x < MLS_CG_SIZE; x++)2866
+; {2867
+; int signCoef = m_resiDctCoeff[blkPos + x]; /* pre-quantization DCT coeff */2868
+; costUncoded[blkPos + x] = static_cast<int64_t>((double)((signCoef * signCoef) << scaleBits));2869
+; *totalUncodedCost += costUncoded[blkPos + x];2870
+; *totalRdCost += costUncoded[blkPos + x];2871
+; }2872
+; blkPos += trSize;2873
+; }2874
+;}2875
+2876
+;---------------------------------------------------------------------------------------------------------------------------------------------------------2877
+; void nonPsyRdoQuant_c(int16_t *m_resiDctCoeff, int64_t *costUncoded, int64_t *totalUncodedCost, int64_t *totalRdCost, uint32_t blkPos)2878
+;---------------------------------------------------------------------------------------------------------------------------------------------------------2879
+INIT_ZMM avx5122880
+cglobal nonPsyRdoQuant4, 5, 5, 82881
+ mov r4d, r4m2882
+ lea r0, [r0 + 2 * r4]2883
+ lea r4, [4 * r4]2884
+ lea r1, [r1 + 2 * r4]2885
+%if BIT_DEPTH == 122886
+ mov r4, [tab_nonpsyRdo12]2887
+%elif BIT_DEPTH == 102888
+ mov r4, [tab_nonpsyRdo10]2889
+%elif BIT_DEPTH == 82890
+ mov r4, [tab_nonpsyRdo8]2891
+%else2892
+ %error Unsupported BIT_DEPTH!2893
+ %endif2894
+ movq xm3, r42895
+ movq xm6, [r2]2896
+ movq xm7, [r3]2897
+ vpxor m4, m42898
+ vpxor m5, m52899
+;Row 1, 22900
+ movu xm0, [r0]2901
+ vpmovsxwq m1, xm02902
+ vcvtqq2pd m2, m1 ; Convert packed 64-bit integers to packed double-precision (64-bit) floating-point elements2903
+ vfmadd213pd m2, m2, m5 ; Multiply packed double-precision (64-bit) floating-point elements2904
+ vcvtpd2qq m1, m22905
+ vpsllq m1, xm3 ; costUncoded2906
+ paddq m4, m12907
+ movu [r1], m12908
+ ;Row 3, 42909
+ movu xm0, [r0 + 16]2910
+ vpmovsxwq m1, xm02911
+ vcvtqq2pd m2, m12912
+ vfmadd213pd m2, m2, m52913
+ vcvtpd2qq m1, m22914
+ vpsllq m1, xm3 ; costUncoded2915
+ paddq m4, m12916
+ movu [r1 + 64], m12917
+ vextracti32x8 ym2, m4, 12918
+ paddq ym4, ym22919
+ vextracti32x4 xm2, m4, 12920
+ paddq xm4, xm22921
+ punpckhqdq xm2, xm4, xm52922
+ paddq xm4, xm22923
+2924
+ paddq xm6, xm42925
+ paddq xm7, xm42926
+2927
+ movq [r2], xm62928
+ movq [r3], xm72929
+ RET2930
+INIT_ZMM avx5122931
+cglobal nonPsyRdoQuant8, 5, 5, 82932
+ mov r4d, r4m2933
+ lea r0, [r0 + 2 * r4]2934
+ lea r4, [4 * r4]2935
+ lea r1, [r1 + 2 * r4]2936
+%if BIT_DEPTH == 122937
+ mov r4, [tab_nonpsyRdo12 + 8]2938
+%elif BIT_DEPTH == 102939
+ mov r4, [tab_nonpsyRdo10 + 8]2940
+%elif BIT_DEPTH == 82941
+ mov r4, [tab_nonpsyRdo8 + 8]2942
+%else2943
+ %error Unsupported BIT_DEPTH!2944
+ %endif2945
+ movq xm3, r42946
+ movq xm6, [r2]2947
+ movq xm7, [r3]2948
+ vpxor m4, m42949
+ vpxor m5, m52950
+2951
+;Row 1, 22952
+ movq xm0, [r0]2953
+ pinsrq xm0, [r0 + mmsize/4], 12954
+ vpmovsxwq m1, xm02955
+ vcvtqq2pd m2, m1 ; Convert packed 64-bit integers to packed double-precision (64-bit) floating-point elements2956
+ vfmadd213pd m2, m2, m5 ; Multiply packed double-precision (64-bit) floating-point elements2957
+ vcvtpd2qq m1, m22958
+ vpsllq m1, xm3 ; costUncoded2959
+ paddq m4, m12960
+ movu [r1], ym12961
+ vextracti32x8 [r1 + mmsize], m1 , 12962
+2963
+ ;Row 3, 42964
+ movq xm0, [r0 + mmsize/2]2965
+ pinsrq xm0, [r0 + 3 * mmsize/4], 12966
+ vpmovsxwq m1, xm02967
+ vcvtqq2pd m2, m12968
+ vfmadd213pd m2, m2, m52969
+ vcvtpd2qq m1, m22970
+ vpsllq m1, xm3 ; costUncoded2971
+ paddq m4, m12972
+ movu [r1 + 2 * mmsize], ym12973
+ vextracti32x8 [r1 + 3 * mmsize], m1 , 12974
+2975
+ vextracti32x8 ym2, m4, 12976
+ paddq ym4, ym22977
+ vextracti32x4 xm2, m4, 12978
+ paddq xm4, xm22979
+ punpckhqdq xm2, xm4, xm52980
+ paddq xm4, xm22981
+2982
+ paddq xm6, xm42983
+ paddq xm7, xm42984
+2985
+ movq [r2], xm62986
+ movq [r3], xm72987
+ RET2988
+INIT_ZMM avx5122989
+cglobal nonPsyRdoQuant16, 5, 5, 82990
+ mov r4d, r4m2991
+ lea r0, [r0 + 2 * r4]2992
+ lea r4, [4 * r4]2993
+ lea r1, [r1 + 2 * r4]2994
+%if BIT_DEPTH == 122995
+ mov r4, [tab_nonpsyRdo12 + 16]2996
+%elif BIT_DEPTH == 102997
+ mov r4, [tab_nonpsyRdo10 + 16]2998
+%elif BIT_DEPTH == 82999
+ mov r4, [tab_nonpsyRdo8 + 16]3000
+%else3001
+ %error Unsupported BIT_DEPTH!3002
+ %endif3003
+ movq xm3, r43004
+ movq xm6, [r2]3005
+ movq xm7, [r3]3006
+ vpxor m4, m43007
+ vpxor m5, m53008
+3009
+;Row 1, 23010
+ movq xm0, [r0]3011
+ pinsrq xm0, [r0 + mmsize/2], 13012
+ vpmovsxwq m1, xm03013
+ vcvtqq2pd m2, m1 ; Convert packed 64-bit integers to packed double-precision (64-bit) floating-point elements3014
+ vfmadd213pd m2, m2, m5 ; Multiply packed double-precision (64-bit) floating-point elements3015
+ vcvtpd2qq m1, m23016
+ vpsllq m1, xm3 ; costUncoded3017
+ paddq m4, m13018
+ movu [r1], ym13019
+ vextracti32x8 [r1 + 2 * mmsize], m1, 13020
+3021
+ ;Row 3, 43022
+ movq xm0, [r0 + mmsize]3023
+ pinsrq xm0, [r0 + 3 * mmsize/2], 13024
+ vpmovsxwq m1, xm03025
+ vcvtqq2pd m2, m13026
+ vfmadd213pd m2, m2, m53027
+ vcvtpd2qq m1, m23028
+ vpsllq m1, xm3 ; costUncoded3029
+ paddq m4, m13030
+ movu [r1 + 4 * mmsize], ym13031
+ vextracti32x8 [r1 + 6 * mmsize], m1 , 13032
+3033
+ vextracti32x8 ym2, m4, 13034
+ paddq ym4, ym23035
+ vextracti32x4 xm2, m4, 13036
+ paddq xm4, xm23037
+ punpckhqdq xm2, xm4, xm53038
+ paddq xm4, xm23039
+3040
+ paddq xm6, xm43041
+ paddq xm7, xm43042
+3043
+ movq [r2], xm63044
+ movq [r3], xm73045
+ RET3046
+INIT_ZMM avx5123047
+cglobal nonPsyRdoQuant32, 5, 5, 83048
+ mov r4d, r4m3049
+ lea r0, [r0 + 2 * r4]3050
+ lea r4, [4 * r4]3051
+ lea r1, [r1 + 2 * r4]3052
+%if BIT_DEPTH == 123053
+ mov r4, [tab_nonpsyRdo12 + 24]3054
+%elif BIT_DEPTH == 103055
+ mov r4, [tab_nonpsyRdo10 + 24]3056
+%elif BIT_DEPTH == 83057
+ mov r4, [tab_nonpsyRdo8 + 24]3058
+%else3059
+ %error Unsupported BIT_DEPTH!3060
+ %endif3061
+ movq xm3, r43062
+ movq xm6, [r2]3063
+ movq xm7, [r3]3064
+ vpxor m4, m43065
+ vpxor m5, m53066
+3067
+;Row 1, 23068
+ movq xm0, [r0]3069
+ pinsrq xm0, [r0 + mmsize], 13070
+ vpmovsxwq m1, xm03071
+ vcvtqq2pd m2, m1 ; Convert packed 64-bit integers to packed double-precision (64-bit) floating-point elements3072
+ vfmadd213pd m2, m2, m5 ; Multiply packed double-precision (64-bit) floating-point elements3073
+ vcvtpd2qq m1, m23074
+ vpsllq m1, xm3 ; costUncoded3075
+ paddq m4, m13076
+ movu [r1], ym13077
+ vextracti32x8 [r1 + 4 * mmsize], m1, 13078
+3079
+ ;Row 3, 43080
+ movq xm0, [r0 + 2 * mmsize]3081
+ pinsrq xm0, [r0 + 3 * mmsize], 13082
+ vpmovsxwq m1, xm03083
+ vcvtqq2pd m2, m13084
+ vfmadd213pd m2, m2, m53085
+ vcvtpd2qq m1, m23086
+ vpsllq m1, xm3 ; costUncoded3087
+ paddq m4, m13088
+ movu [r1 + 8 * mmsize], ym13089
+ vextracti32x8 [r1 + 12 * mmsize], m1 , 13090
+3091
+ vextracti32x8 ym2, m4, 13092
+ paddq ym4, ym23093
+ vextracti32x4 xm2, m4, 13094
+ paddq xm4, xm23095
+ punpckhqdq xm2, xm4, xm53096
+ paddq xm4, xm23097
+3098
+ paddq xm6, xm43099
+ paddq xm7, xm43100
+3101
+ movq [r2], xm63102
+ movq [r3], xm73103
+ RET3104
+;static void psyRdoQuant_c(int16_t *m_resiDctCoeff, int16_t *m_fencDctCoeff, int64_t *costUncoded, int64_t *totalUncodedCost, int64_t *totalRdCost, int64_t psyScale, uint32_t blkPos)3105
+;{3106
+; const int transformShift = MAX_TR_DYNAMIC_RANGE - X265_DEPTH - log2TrSize; /* Represents scaling through forward transform */3107
+; const int scaleBits = SCALE_BITS - 2 * transformShift;3108
+; const uint32_t trSize = 1 << log2TrSize;3109
+; int max = X265_MAX(0, (2 * transformShift + 1));3110
+;3111
+; for (int y = 0; y < MLS_CG_SIZE; y++)3112
+; {3113
+; for (int x = 0; x < MLS_CG_SIZE; x++)3114
+; {3115
+; int64_t signCoef = m_resiDctCoeff[blkPos + x]; /* pre-quantization DCT coeff */3116
+; int64_t predictedCoef = m_fencDctCoeff[blkPos + x] - signCoef; /* predicted DCT = source DCT - residual DCT*/3117
+;3118
+; costUncoded[blkPos + x] = static_cast<int64_t>((double)(signCoef * signCoef) << scaleBits);3119
+;3120
+; /* when no residual coefficient is coded, predicted coef == recon coef */3121
+; costUncoded[blkPos + x] -= static_cast<int64_t>((psyScale * (predictedCoef)) >> max);3122
+;3123
+; *totalUncodedCost += costUncoded[blkPos + x];3124
+; *totalRdCost += costUncoded[blkPos + x];3125
+; }3126
+; blkPos += trSize;3127
+; }3128
+;}3129
+3130
+;---------------------------------------------------------------------------------------------------------------------------------------------------------3131
+; void psyRdoQuant(int16_t *m_resiDctCoeff, int16_t *m_fencDctCoeff, int64_t *costUncoded, int64_t *totalUncodedCost, int64_t *totalRdCost, int64_t *psyScale, uint32_t blkPos)3132
+;---------------------------------------------------------------------------------------------------------------------------------------------------------3133
+INIT_ZMM avx5123134
+cglobal psyRdoQuant4, 5, 9, 133135
+%if WIN643136
+ mov r5, r5m3137
+%endif3138
+ mov r6d, r6m3139
+ vpbroadcastq m12, [r5] ; psyScale3140
+ lea r0, [r0 + 2 * r6]3141
+ lea r1, [r1 + 2 * r6]3142
+ lea r6, [4 * r6]3143
+ lea r2, [r2 + 2 * r6]3144
+ movq xm0, [r3]3145
+ movq xm1, [r4]3146
+3147
+%if BIT_DEPTH == 123148
+ mov r5, [tab_nonpsyRdo12] ; scaleBits3149
+%elif BIT_DEPTH == 103150
+ mov r5, [tab_nonpsyRdo10]3151
+%elif BIT_DEPTH == 83152
+ mov r5, [tab_nonpsyRdo8]3153
+%else3154
+ %error Unsupported BIT_DEPTH!3155
+%endif3156
+3157
+ movq xm2, r53158
+ vpxor m4, m43159
+ vpxor m3, m33160
+3161
+;Row 1, 23162
+ vpmovsxwq m6, [r0]3163
+ vpmovsxwq m7, [r1]3164
+ psubq m7, m6 ; predictedCoef3165
+3166
+ vcvtqq2pd m9, m63167
+ vfmadd213pd m9, m9, m33168
+ vcvtpd2qq m8, m93169
+ vpsllq m8, xm2 ;(signCoef * signCoef) << scaleBits3170
+3171
+ vcvtqq2pd m10, m73172
+ vcvtqq2pd m11, m123173
+ vfmadd213pd m10, m11, m33174
+ vcvtpd2qq m9, m103175
+ vpsraq m9, RDO_MAX_4 ;(psyScale * predictedCoef) >> max3176
+3177
+ psubq m8, m93178
+ paddq m4, m83179
+ movu [r2], m83180
+3181
+ ;Row 3, 43182
+ vpmovsxwq m6, [r0 + 16]3183
+ vpmovsxwq m7, [r1 + 16]3184
+ psubq m7, m6 ; predictedCoef3185
+3186
+ vcvtqq2pd m9, m63187
+ vfmadd213pd m9, m9, m33188
+ vcvtpd2qq m8, m93189
+ vpsllq m8, xm2 ;(signCoef * signCoef) << scaleBits3190
+3191
+ vcvtqq2pd m10, m73192
+ vcvtqq2pd m11, m123193
+ vfmadd213pd m10, m11, m33194
+ vcvtpd2qq m9, m103195
+ vpsraq m9, RDO_MAX_4 ;(psyScale * predictedCoef) >> max3196
+3197
+ psubq m8, m93198
+ paddq m4, m83199
+ movu [r2 + 64], m83200
+3201
+ vextracti32x8 ym2, m4, 13202
+ paddq ym4, ym23203
+ vextracti32x4 xm2, m4, 13204
+ paddq xm4, xm23205
+ punpckhqdq xm2, xm4, xm33206
+ paddq xm4, xm23207
+3208
+ paddq xm0, xm43209
+ paddq xm1, xm43210
+3211
+ movq [r3], xm03212
+ movq [r4], xm13213
+ RET3214
+3215
+;---------------------------------------------------------------------------------------------------------------------------------------------------------3216
+; void psyRdoQuant(int16_t *m_resiDctCoeff, int16_t *m_fencDctCoeff, int64_t *costUncoded, int64_t *totalUncodedCost, int64_t *totalRdCost, int64_t *psyScale, uint32_t blkPos)3217
+;---------------------------------------------------------------------------------------------------------------------------------------------------------3218
+INIT_ZMM avx5123219
+cglobal psyRdoQuant8, 5, 9, 153220
+%if WIN643221
+ mov r5, r5m3222
+%endif3223
+ mov r6d, r6m3224
+ vpbroadcastq m12, [r5] ; psyScale3225
+ lea r0, [r0 + 2 * r6]3226
+ lea r1, [r1 + 2 * r6]3227
+ lea r6, [4 * r6]3228
+ lea r2, [r2 + 2 * r6]3229
+ movq xm0, [r3]3230
+ movq xm1, [r4]3231
+3232
+%if BIT_DEPTH == 123233
+ mov r5, [tab_nonpsyRdo12 + 8] ; scaleBits3234
+%elif BIT_DEPTH == 103235
+ mov r5, [tab_nonpsyRdo10 + 8]3236
+%elif BIT_DEPTH == 83237
+ mov r5, [tab_nonpsyRdo8 + 8]3238
+%else3239
+ %error Unsupported BIT_DEPTH!3240
+%endif3241
+3242
+ movq xm2, r53243
+ vpxor m4, m43244
+ vpxor m3, m33245
+3246
+;Row 1, 23247
+ movq xm13, [r0]3248
+ movq xm14, [r1]3249
+ pinsrq xm13, [r0 + mmsize/4], 13250
+ pinsrq xm14, [r1 + mmsize/4], 13251
+ vpmovsxwq m6, xm133252
+ vpmovsxwq m7, xm143253
+ psubq m7, m6 ; predictedCoef3254
+3255
+ vcvtqq2pd m9, m63256
+ vfmadd213pd m9, m9, m33257
+ vcvtpd2qq m8, m93258
+ vpsllq m8, xm2 ;(signCoef * signCoef) << scaleBits3259
+3260
+ vcvtqq2pd m10, m73261
+ vcvtqq2pd m11, m123262
+ vfmadd213pd m10, m11, m33263
+ vcvtpd2qq m9, m103264
+ vpsraq m9, RDO_MAX_8 ;(psyScale * predictedCoef) >> max3265
+3266
+ psubq m8, m93267
+ paddq m4, m83268
+ movu [r2], ym83269
+ vextracti32x8 [r2 + mmsize], m8 , 13270
+3271
+ ;Row 3, 43272
+ movq xm13, [r0 + mmsize/2]3273
+ movq xm14, [r1 + mmsize/2]3274
+ pinsrq xm13, [r0 + 3 * mmsize/4], 13275
+ pinsrq xm14, [r1 + 3 * mmsize/4], 13276
+ vpmovsxwq m6, xm133277
+ vpmovsxwq m7, xm143278
+ psubq m7, m6 ; predictedCoef3279
+3280
+ vcvtqq2pd m9, m63281
+ vfmadd213pd m9, m9, m33282
+ vcvtpd2qq m8, m93283
+ vpsllq m8, xm2 ;(signCoef * signCoef) << scaleBits3284
+3285
+ vcvtqq2pd m10, m73286
+ vcvtqq2pd m11, m123287
+ vfmadd213pd m10, m11, m33288
+ vcvtpd2qq m9, m103289
+ vpsraq m9, RDO_MAX_8 ;(psyScale * predictedCoef) >> max3290
+3291
+ psubq m8, m93292
+ paddq m4, m83293
+ movu [r2 + 2 * mmsize], ym83294
+ vextracti32x8 [r2 + 3 * mmsize], m8 , 13295
+3296
+ vextracti32x8 ym2, m4, 13297
+ paddq ym4, ym23298
+ vextracti32x4 xm2, m4, 13299
+ paddq xm4, xm23300
+ punpckhqdq xm2, xm4, xm33301
+ paddq xm4, xm23302
+3303
+ paddq xm0, xm43304
+ paddq xm1, xm43305
+3306
+ movq [r3], xm03307
+ movq [r4], xm13308
+ RET3309
+3310
+;---------------------------------------------------------------------------------------------------------------------------------------------------------3311
+; void psyRdoQuant(int16_t *m_resiDctCoeff, int16_t *m_fencDctCoeff, int64_t *costUncoded, int64_t *totalUncodedCost, int64_t *totalRdCost, int64_t *psyScale, uint32_t blkPos)3312
+;---------------------------------------------------------------------------------------------------------------------------------------------------------3313
+INIT_ZMM avx5123314
+cglobal psyRdoQuant16, 5, 9, 153315
+%if WIN643316
+ mov r5, r5m3317
+%endif3318
+ mov r6d, r6m3319
+ vpbroadcastq m12, [r5] ; psyScale3320
+ lea r0, [r0 + 2 * r6]3321
+ lea r1, [r1 + 2 * r6]3322
+ lea r6, [4 * r6]3323
+ lea r2, [r2 + 2 * r6]3324
+ movq xm0, [r3]3325
+ movq xm1, [r4]3326
+3327
+%if BIT_DEPTH == 123328
+ mov r5, [tab_nonpsyRdo12 + 16] ; scaleBits3329
+%elif BIT_DEPTH == 103330
+ mov r5, [tab_nonpsyRdo10 + 16]3331
+%elif BIT_DEPTH == 83332
+ mov r5, [tab_nonpsyRdo8 + 16]3333
+%else3334
+ %error Unsupported BIT_DEPTH!3335
+%endif3336
+3337
+ movq xm2, r53338
+ vpxor m4, m43339
+ vpxor m3, m33340
+3341
+;Row 1, 23342
+ movq xm13, [r0]3343
+ movq xm14, [r1]3344
+ pinsrq xm13, [r0 + mmsize/2], 13345
+ pinsrq xm14, [r1 + mmsize/2], 13346
+ vpmovsxwq m6, xm133347
+ vpmovsxwq m7, xm143348
+ psubq m7, m6 ; predictedCoef3349
+3350
+ vcvtqq2pd m9, m63351
+ vfmadd213pd m9, m9, m33352
+ vcvtpd2qq m8, m93353
+ vpsllq m8, xm2 ;(signCoef * signCoef) << scaleBits3354
+3355
+ vcvtqq2pd m10, m73356
+ vcvtqq2pd m11, m123357
+ vfmadd213pd m10, m11, m33358
+ vcvtpd2qq m9, m103359
+ vpsraq m9, RDO_MAX_16 ;(psyScale * predictedCoef) >> max3360
+3361
+ psubq m8, m93362
+ paddq m4, m83363
+ movu [r2], ym83364
+ vextracti32x8 [r2 + 2 * mmsize], m8 , 13365
+3366
+ ;Row 3, 43367
+ movq xm13, [r0 + mmsize]3368
+ movq xm14, [r1 + mmsize]3369
+ pinsrq xm13, [r0 + 3 * mmsize/2], 13370
+ pinsrq xm14, [r1 + 3 * mmsize/2], 13371
+ vpmovsxwq m6, xm133372
+ vpmovsxwq m7, xm143373
+ psubq m7, m6 ; predictedCoef3374
+3375
+ vcvtqq2pd m9, m63376
+ vfmadd213pd m9, m9, m33377
+ vcvtpd2qq m8, m93378
+ vpsllq m8, xm2 ;(signCoef * signCoef) << scaleBits3379
+3380
+ vcvtqq2pd m10, m73381
+ vcvtqq2pd m11, m123382
+ vfmadd213pd m10, m11, m33383
+ vcvtpd2qq m9, m103384
+ vpsraq m9, RDO_MAX_16 ;(psyScale * predictedCoef) >> max3385
+3386
+ psubq m8, m93387
+ paddq m4, m83388
+ movu [r2 + 4 * mmsize], ym83389
+ vextracti32x8 [r2 + 6 * mmsize], m8 , 13390
+3391
+ vextracti32x8 ym2, m4, 13392
+ paddq ym4, ym23393
+ vextracti32x4 xm2, m4, 13394
+ paddq xm4, xm23395
+ punpckhqdq xm2, xm4, xm33396
+ paddq xm4, xm23397
+3398
+ paddq xm0, xm43399
+ paddq xm1, xm43400
+3401
+ movq [r3], xm03402
+ movq [r4], xm13403
+ RET3404
+3405
+;---------------------------------------------------------------------------------------------------------------------------------------------------------3406
+; void psyRdoQuant(int16_t *m_resiDctCoeff, int16_t *m_fencDctCoeff, int64_t *costUncoded, int64_t *totalUncodedCost, int64_t *totalRdCost, int64_t *psyScale, uint32_t blkPos)3407
+;---------------------------------------------------------------------------------------------------------------------------------------------------------3408
+INIT_ZMM avx5123409
+cglobal psyRdoQuant32, 5, 9, 153410
+%if WIN643411
+ mov r5, r5m3412
+%endif3413
+ mov r6d, r6m3414
+ vpbroadcastq m12, [r5] ; psyScale3415
+ lea r0, [r0 + 2 * r6]3416
+ lea r1, [r1 + 2 * r6]3417
+ lea r6, [4 * r6]3418
+ lea r2, [r2 + 2 * r6]3419
+ movq xm0, [r3]3420
+ movq xm1, [r4]3421
+3422
+%if BIT_DEPTH == 123423
+ mov r5, [tab_nonpsyRdo12 + 24] ; scaleBits3424
+%elif BIT_DEPTH == 103425
+ mov r5, [tab_nonpsyRdo10 + 24]3426
+%elif BIT_DEPTH == 83427
+ mov r5, [tab_nonpsyRdo8 + 24]3428
+%else3429
+ %error Unsupported BIT_DEPTH!3430
+%endif3431
+3432
+ movq xm2, r53433
+ vpxor m4, m43434
+ vpxor m3, m33435
+3436
+;Row 1, 23437
+ movq xm13, [r0]3438
+ movq xm14, [r1]3439
+ pinsrq xm13, [r0 + mmsize], 13440
+ pinsrq xm14, [r1 + mmsize], 13441
+ vpmovsxwq m6, xm133442
+ vpmovsxwq m7, xm143443
+ psubq m7, m6 ; predictedCoef3444
+3445
+ vcvtqq2pd m9, m63446
+ vfmadd213pd m9, m9, m33447
+ vcvtpd2qq m8, m93448
+ vpsllq m8, xm2 ;(signCoef * signCoef) << scaleBits3449
+3450
+ vcvtqq2pd m10, m73451
+ vcvtqq2pd m11, m123452
+ vfmadd213pd m10, m11, m33453
+ vcvtpd2qq m9, m103454
+ vpsraq m9, RDO_MAX_32 ;(psyScale * predictedCoef) >> max3455
+3456
+ psubq m8, m93457
+ paddq m4, m83458
+ movu [r2], ym83459
+ vextracti32x8 [r2 + 4 * mmsize], m8 , 13460
+3461
+ ;Row 3, 43462
+ movq xm13, [r0 + 2 * mmsize]3463
+ movq xm14, [r1 + 2 * mmsize]3464
+ pinsrq xm13, [r0 + 3 * mmsize], 13465
+ pinsrq xm14, [r1 + 3 * mmsize], 13466
+ vpmovsxwq m6, xm133467
+ vpmovsxwq m7, xm143468
+ psubq m7, m6 ; predictedCoef3469
+3470
+ vcvtqq2pd m9, m63471
+ vfmadd213pd m9, m9, m33472
+ vcvtpd2qq m8, m93473
+ vpsllq m8, xm2 ;(signCoef * signCoef) << scaleBits3474
+3475
+ vcvtqq2pd m10, m73476
+ vcvtqq2pd m11, m123477
+ vfmadd213pd m10, m11, m33478
+ vcvtpd2qq m9, m103479
+ vpsraq m9, RDO_MAX_32 ;(psyScale * predictedCoef) >> max3480
+3481
+ psubq m8, m93482
+ paddq m4, m83483
+ movu [r2 + 8 * mmsize], ym83484
+ vextracti32x8 [r2 + 12 * mmsize], m8 , 13485
+3486
+ vextracti32x8 ym2, m4, 13487
+ paddq ym4, ym23488
+ vextracti32x4 xm2, m4, 13489
+ paddq xm4, xm23490
+ punpckhqdq xm2, xm4, xm33491
+ paddq xm4, xm23492
+3493
+ paddq xm0, xm43494
+ paddq xm1, xm43495
+3496
+ movq [r3], xm03497
+ movq [r4], xm13498
+ RET3499
+3500
+INIT_YMM avx23501
+cglobal nonPsyRdoQuant4, 5, 9, 163502
+ mov r4d, r4m3503
+ lea r0, [r0 + 2 * r4]3504
+ lea r4, [4 * r4]3505
+ lea r1, [r1 + 2 * r4]3506
+ movq xm0, [r2]3507
+ movq xm1, [r3]3508
+3509
+%if BIT_DEPTH == 123510
+ mov r5, [tab_nonpsyRdo12] ; scaleBits3511
+%elif BIT_DEPTH == 103512
+ mov r5, [tab_nonpsyRdo10]3513
+%elif BIT_DEPTH == 83514
+ mov r5, [tab_nonpsyRdo8]3515
+%else3516
+ %error Unsupported BIT_DEPTH!3517
+%endif3518
+ movq xm2, r53519
+ vpxor m4, m43520
+ vpxor m3, m33521
+ vpxor m13, m133522
+3523
+ vpmovsxwd m6, [r0]3524
+ vcvtdq2pd m9, xm63525
+ vfmadd213pd m9, m9, m33526
+ vcvtpd2dq xm8, m93527
+ vpmovsxdq m13, xm8 ; 32 bit int to 64 bit int3528
+ vpsllq m13, xm2 ;(signCoef * signCoef) << scaleBits 3529
+ paddq m4, m133530
+ movu [r1], m133531
+3532
+ vpmovsxwd m6, [r0 + 8]3533
+ vcvtdq2pd m9, xm63534
+ vfmadd213pd m9, m9, m33535
+ vcvtpd2dq xm8, m93536
+ vpmovsxdq m13, xm8 ; 32 bit int to 64 bit int3537
+ vpsllq m13, xm2 ;(signCoef * signCoef) << scaleBits 3538
+ paddq m4, m133539
+ movu [r1 + 32], m133540
+3541
+ vpmovsxwd m6, [r0 + 16]3542
+ vcvtdq2pd m9, xm63543
+ vfmadd213pd m9, m9, m33544
+ vcvtpd2dq xm8, m93545
+ vpmovsxdq m13, xm8 ; 32 bit int to 64 bit int3546
+ vpsllq m13, xm2 ;(signCoef * signCoef) << scaleBits 3547
+ paddq m4, m133548
+ movu [r1 + 64], m133549
+3550
+ vpmovsxwd m6, [r0 +24]3551
+ vcvtdq2pd m9, xm63552
+ vfmadd213pd m9, m9, m33553
+ vcvtpd2dq xm8, m93554
+ vpmovsxdq m13, xm8 ; 32 bit int to 64 bit int 3555
+ vpsllq m13, xm2 ;(signCoef * signCoef) << scaleBits3556
+ paddq m4, m133557
+ movu [r1 + 96], m133558
+3559
+3560
+ vextracti128 xm2, m4, 13561
+ paddq xm4, xm23562
+ punpckhqdq xm2, xm4, xm33563
+ paddq xm4, xm23564
+3565
+ paddq xm0, xm43566
+ paddq xm1, xm43567
+3568
+ movq [r2], xm03569
+ movq [r3], xm13570
+ RET3571
+3572
+3573
+3574
+INIT_YMM avx23575
+cglobal nonPsyRdoQuant8, 5, 5, 83576
+ mov r4d, r4m3577
+ lea r0, [r0 + 2 * r4]3578
+ lea r4, [4 * r4]3579
+ lea r1, [r1 + 2 * r4]3580
+%if BIT_DEPTH == 123581
+ mov r4, [tab_nonpsyRdo12 + 8]3582
+%elif BIT_DEPTH == 103583
+ mov r4, [tab_nonpsyRdo10 + 8]3584
+%elif BIT_DEPTH == 83585
+ mov r4, [tab_nonpsyRdo8 + 8]3586
+%else3587
+ %error Unsupported BIT_DEPTH!3588
+ %endif3589
+ movq xm3, r43590
+ movq xm6, [r2]3591
+ movq xm7, [r3]3592
+ vpxor m4, m43593
+ vpxor m5, m53594
+ movq xm0, [r0]3595
+ vpmovsxwd m1, xm03596
+ vcvtdq2pd m2, xm1 ; Convert packed 64-bit integers to packed double-precision (64-bit) floating-point elements3597
+ vfmadd213pd m2, m2, m5 ; Multiply packed double-precision (64-bit) floating-point elements3598
+ vcvtpd2dq xm1, m23599
+ vpmovsxdq m0 , xm13600
+ vpsllq m0, xm3 ; costUncoded3601
+ paddq m4, m03602
+ movu [r1], ym03603
+ vpxor m0, m03604
+ movq xm0, [r0 +mmsize/2]3605
+ vpmovsxwd m1, xm03606
+ vcvtdq2pd m2, xm1 ; Convert packed 64-bit integers to packed double-precision (64-bit) floating-point elements3607
+ vfmadd213pd m2, m2, m5 ; Multiply packed double-precision (64-bit) floating-point elements3608
+ vcvtpd2dq xm1, m23609
+ vpmovsxdq m0 , xm13610
+ vpsllq m0, xm3 ; costUncoded3611
+ paddq m4, m03612
+ movu [r1 +2*mmsize], m03613
+ vpxor m0, m03614
+ movq xm0, [r0 +mmsize]3615
+ vpmovsxwd m1, xm03616
+ vcvtdq2pd m2, xm1 ; Convert packed 64-bit integers to packed double-precision (64-bit) floating-point elements3617
+ vfmadd213pd m2, m2, m5 ; Multiply packed double-precision (64-bit) floating-point elements3618
+ vcvtpd2dq xm1, m23619
+ vpmovsxdq m0 , xm13620
+ vpsllq m0, xm3 ; costUncoded3621
+ paddq m4, m03622
+ movu [r1 +4*mmsize], m03623
+ vpxor m0, m03624
+ movq xm0, [r0 +3*mmsize/2]3625
+ vpmovsxwd m1, xm03626
+ vcvtdq2pd m2, xm1 ; Convert packed 64-bit integers to packed double-precision (64-bit) floating-point elements3627
+ vfmadd213pd m2, m2, m5 ; Multiply packed double-precision (64-bit) floating-point elements3628
+ vcvtpd2dq xm1, m23629
+ vpmovsxdq m0 , xm13630
+ vpsllq m0, xm3 ; costUncoded3631
+ paddq m4, m03632
+ movu [r1 +6*mmsize], m03633
+3634
+ vextracti128 xm2, m4, 13635
+ paddq xm4, xm23636
+ punpckhqdq xm2, xm4, xm53637
+ paddq xm4, xm23638
+3639
+ paddq xm6, xm43640
+ paddq xm7, xm43641
+3642
+ movq [r2], xm63643
+ movq [r3], xm73644
+ RET3645
+INIT_YMM avx23646
+cglobal nonPsyRdoQuant16, 5, 5, 83647
+ mov r4d, r4m3648
+ lea r0, [r0 + 2 * r4]3649
+ lea r4, [4 * r4]3650
+ lea r1, [r1 + 2 * r4]3651
+%if BIT_DEPTH == 123652
+ mov r4, [tab_nonpsyRdo12 + 16]3653
+%elif BIT_DEPTH == 103654
+ mov r4, [tab_nonpsyRdo10 + 16]3655
+%elif BIT_DEPTH == 83656
+ mov r4, [tab_nonpsyRdo8 + 16]3657
+%else3658
+ %error Unsupported BIT_DEPTH!3659
+ %endif3660
+ movq xm3, r43661
+ movq xm6, [r2]3662
+ movq xm7, [r3]3663
+ vpxor m4, m43664
+ vpxor m5, m53665
+3666
+;Row 1, 23667
+ movq xm0, [r0]3668
+ vpmovsxwd m1, xm03669
+ vcvtdq2pd m2, xm1 ; Convert packed 64-bit integers to packed double-precision (64-bit) floating-point elements3670
+ vfmadd213pd m2, m2, m5 ; Multiply packed double-precision (64-bit) floating-point elements3671
+ vcvtpd2dq xm1, m23672
+ vpmovsxdq m0 , xm13673
+ vpsllq m0, xm3 ; costUncoded3674
+ paddq m4, m03675
+ movu [r1], ym03676
+ 3677
+ movq xm0, [r0 +mmsize]3678
+ vpmovsxwd m1, xm03679
+ vcvtdq2pd m2, xm1 ; Convert packed 64-bit integers to packed double-precision (64-bit) floating-point elements3680
+ vfmadd213pd m2, m2, m5 ; Multiply packed double-precision (64-bit) floating-point elements3681
+ vcvtpd2dq xm1, m23682
+ vpmovsxdq m0 , xm13683
+ vpsllq m0, xm3 ; costUncoded3684
+ paddq m4, m03685
+ movu [r1+4*mmsize], ym03686
+ 3687
+ movq xm0, [r0 + 2*mmsize]3688
+ vpmovsxwd m1, xm03689
+ vcvtdq2pd m2, xm1 ; Convert packed 64-bit integers to packed double-precision (64-bit) floating-point elements3690
+ vfmadd213pd m2, m2, m5 ; Multiply packed double-precision (64-bit) floating-point elements3691
+ vcvtpd2dq xm1, m23692
+ vpmovsxdq m0 , xm13693
+ vpsllq m0, xm3 ; costUncoded3694
+ paddq m4, m03695
+ movu [r1+8*mmsize], ym03696
+3697
+ movq xm0, [r0 + 3*mmsize]3698
+ vpmovsxwd m1, xm03699
+ vcvtdq2pd m2, xm1 ; Convert packed 64-bit integers to packed double-precision (64-bit) floating-point elements3700
+ vfmadd213pd m2, m2, m5 ; Multiply packed double-precision (64-bit) floating-point elements3701
+ vcvtpd2dq xm1, m23702
+ vpmovsxdq m0 , xm13703
+ vpsllq m0, xm3 ; costUncoded3704
+ paddq m4, m03705
+ movu [r1+12*mmsize], ym03706
+3707
+ 3708
+ vextracti128 xm2, m4, 13709
+ paddq xm4, xm23710
+ punpckhqdq xm2, xm4, xm53711
+ paddq xm4, xm23712
+3713
+ paddq xm6, xm43714
+ paddq xm7, xm43715
+3716
+ movq [r2], xm63717
+ movq [r3], xm73718
+ RET3719
+INIT_YMM avx23720
+cglobal nonPsyRdoQuant32, 5, 5, 83721
+ mov r4d, r4m3722
+ lea r0, [r0 + 2 * r4]3723
+ lea r4, [4 * r4]3724
+ lea r1, [r1 + 2 * r4]3725
+%if BIT_DEPTH == 123726
+ mov r4, [tab_nonpsyRdo12 + 24]3727
+%elif BIT_DEPTH == 103728
+ mov r4, [tab_nonpsyRdo10 + 24]3729
+%elif BIT_DEPTH == 83730
+ mov r4, [tab_nonpsyRdo8 + 24]3731
+%else3732
+ %error Unsupported BIT_DEPTH!3733
+ %endif3734
+ movq xm3, r43735
+ movq xm6, [r2]3736
+ movq xm7, [r3]3737
+ vpxor m4, m43738
+ vpxor m5, m53739
+3740
+ movq xm0, [r0]3741
+ vpmovsxwd m1, xm03742
+ vcvtdq2pd m2, xm1 ; Convert packed 64-bit integers to packed double-precision (64-bit) floating-point elements3743
+ vfmadd213pd m2, m2, m5 ; Multiply packed double-precision (64-bit) floating-point elements3744
+ vcvtpd2dq xm1, m23745
+ vpmovsxdq m0 , xm13746
+ vpsllq m0, xm3 ; costUncoded3747
+ paddq m4, m03748
+ movu [r1], m03749
+ vpxor m0, m03750
+ 3751
+ movq xm0, [r0 +2*mmsize]3752
+ vpmovsxwd m1, xm03753
+ vcvtdq2pd m2, xm1 ; Convert packed 64-bit integers to packed double-precision (64-bit) floating-point elements3754
+ vfmadd213pd m2, m2, m5 ; Multiply packed double-precision (64-bit) floating-point elements3755
+ vcvtpd2dq xm1, m23756
+ vpmovsxdq m0 , xm13757
+ vpsllq m0, xm3 ; costUncoded3758
+ paddq m4, m03759
+ movu [r1 + 8*mmsize], m03760
+ vpxor m0, m03761
+ 3762
+ movq xm0, [r0 +4*mmsize]3763
+ vpmovsxwd m1, xm03764
+ vcvtdq2pd m2, xm1 ; Convert packed 64-bit integers to packed double-precision (64-bit) floating-point elements3765
+ vfmadd213pd m2, m2, m5 ; Multiply packed double-precision (64-bit) floating-point elements3766
+ vcvtpd2dq xm1, m23767
+ vpmovsxdq m0 , xm13768
+ vpsllq m0, xm3 ; costUncoded3769
+ paddq m4, m03770
+ movu [r1 +16*mmsize], m03771
+ vpxor m0, m03772
+3773
+ movq xm0, [r0 +6*mmsize]3774
+ vpmovsxwd m1, xm03775
+ vcvtdq2pd m2, xm1 ; Convert packed 64-bit integers to packed double-precision (64-bit) floating-point elements3776
+ vfmadd213pd m2, m2, m5 ; Multiply packed double-precision (64-bit) floating-point elements3777
+ vcvtpd2dq xm1, m23778
+ vpmovsxdq m0 , xm13779
+ vpsllq m0, xm3 ; costUncoded3780
+ paddq m4, m03781
+ movu [r1 +24*mmsize], m03782
+3783
+ vextracti128 xm2, m4, 13784
+ paddq xm4, xm23785
+ punpckhqdq xm2, xm4, xm53786
+ paddq xm4, xm23787
+3788
+ paddq xm6, xm43789
+ paddq xm7, xm43790
+3791
+ movq [r2], xm63792
+ movq [r3], xm73793
+ RET3794
+3795
+INIT_YMM avx23796
+cglobal psyRdoQuant_1p4, 5, 9, 163797
+ mov r4d, r4m3798
+ lea r0, [r0 + 2 * r4]3799
+ lea r4, [4 * r4]3800
+ lea r1, [r1 + 2 * r4]3801
+ movq xm0, [r2]3802
+ movq xm1, [r3]3803
+3804
+%if BIT_DEPTH == 123805
+ mov r5, [tab_nonpsyRdo12] ; scaleBits3806
+%elif BIT_DEPTH == 103807
+ mov r5, [tab_nonpsyRdo10]3808
+%elif BIT_DEPTH == 83809
+ mov r5, [tab_nonpsyRdo8]3810
+%else3811
+ %error Unsupported BIT_DEPTH!3812
+%endif3813
+ movq xm2, r53814
+ vpxor m4, m43815
+ vpxor m3, m33816
+ vpxor m13, m133817
+3818
+ vpmovsxwd m6, [r0]3819
+ vcvtdq2pd m9, xm63820
+ vfmadd213pd m9, m9, m33821
+ vcvtpd2dq xm8, m93822
+ vpmovsxdq m13, xm8 ; 32 bit int to 64 bit int3823
+ vpsllq m13, xm2 ;(signCoef * signCoef) << scaleBits 3824
+ paddq m4, m133825
+ movu [r1], m133826
+3827
+ vpmovsxwd m6, [r0 + 8]3828
+ vcvtdq2pd m9, xm63829
+ vfmadd213pd m9, m9, m33830
+ vcvtpd2dq xm8, m93831
+ vpmovsxdq m13, xm8 ; 32 bit int to 64 bit int3832
+ vpsllq m13, xm2 ;(signCoef * signCoef) << scaleBits 3833
+ paddq m4, m133834
+ movu [r1 + 32], m133835
+3836
+ vpmovsxwd m6, [r0 + 16]3837
+ vcvtdq2pd m9, xm63838
+ vfmadd213pd m9, m9, m33839
+ vcvtpd2dq xm8, m93840
+ vpmovsxdq m13, xm8 ; 32 bit int to 64 bit int3841
+ vpsllq m13, xm2 ;(signCoef * signCoef) << scaleBits 3842
+ paddq m4, m133843
+ movu [r1 + 64], m133844
+3845
+ vpmovsxwd m6, [r0 +24]3846
+ vcvtdq2pd m9, xm63847
+ vfmadd213pd m9, m9, m33848
+ vcvtpd2dq xm8, m93849
+ vpmovsxdq m13, xm8 ; 32 bit int to 64 bit int 3850
+ vpsllq m13, xm2 ;(signCoef * signCoef) << scaleBits3851
+ paddq m4, m133852
+ movu [r1 + 96], m133853
+3854
+3855
+ vextracti128 xm2, m4, 13856
+ paddq xm4, xm23857
+ punpckhqdq xm2, xm4, xm33858
+ paddq xm4, xm23859
+3860
+ paddq xm0, xm43861
+ paddq xm1, xm43862
+3863
+ movq [r2], xm03864
+ movq [r3], xm13865
+ RET3866
+INIT_YMM avx23867
+cglobal psyRdoQuant_1p8, 7, 9, 163868
+ mov r4d, r4m3869
+ lea r0, [r0 + 2 * r4]3870
+ lea r4, [4 * r4]3871
+ lea r1, [r1 + 2 * r4]3872
+ movq xm0, [r2]3873
+ movq xm1, [r3]3874
+%if BIT_DEPTH == 123875
+ mov r5, [tab_nonpsyRdo12 +8] ; scaleBits3876
+%elif BIT_DEPTH == 103877
+ mov r5, [tab_nonpsyRdo10 +8]3878
+%elif BIT_DEPTH == 83879
+ mov r5, [tab_nonpsyRdo8 + 8 ]3880
+%else3881
+ %error Unsupported BIT_DEPTH!3882
+%endif3883
+ movq xm2, r53884
+ vpxor m4, m43885
+ vpxor m3, m33886
+ vpxor m13, m133887
+3888
+3889
+ vpmovsxwd m6, [r0]3890
+ vcvtdq2pd m9, xm63891
+ vfmadd213pd m9, m9, m33892
+ vcvtpd2dq xm8, m93893
+ vpmovsxdq m13, xm8 ; 32 bit int to 64 bit int3894
+ vpsllq m13, xm2 ;(signCoef * signCoef) << scaleBits 3895
+ paddq m4, m133896
+ movu [r1], m133897
+3898
+ vpmovsxwd m6, [r0 + 16]3899
+ vcvtdq2pd m9, xm63900
+ vfmadd213pd m9, m9, m33901
+ vcvtpd2dq xm8, m93902
+ vpmovsxdq m13, xm8 ; 32 bit int to 64 bit int3903
+ vpsllq m13, xm2 ;(signCoef * signCoef) << scaleBits 3904
+ paddq m4, m133905
+ movu [r1 + 64], m133906
+3907
+ vpmovsxwd m6, [r0 +32]3908
+ vcvtdq2pd m9, xm63909
+ vfmadd213pd m9, m9, m33910
+ vcvtpd2dq xm8, m93911
+ vpmovsxdq m13, xm8 ; 32 bit int to 64 bit int3912
+ vpsllq m13, xm2 ;(signCoef * signCoef) << scaleBits 3913
+ paddq m4, m133914
+ movu [r1 +128], m133915
+3916
+ vpmovsxwd m6, [r0 + 48]3917
+ vcvtdq2pd m9, xm63918
+ vfmadd213pd m9, m9, m33919
+ vcvtpd2dq xm8, m93920
+ vpmovsxdq m13, xm8 ; 32 bit int to 64 bit int3921
+ vpsllq m13, xm2 ;(signCoef * signCoef) << scaleBits 3922
+ paddq m4, m133923
+ movu [r1 + 192], m133924
+3925
+ vextracti128 xm2, m4, 13926
+ paddq xm4, xm23927
+ punpckhqdq xm2, xm4, xm33928
+ paddq xm4, xm23929
+3930
+ paddq xm0, xm43931
+ paddq xm1, xm43932
+3933
+ movq [r2], xm03934
+ movq [r3], xm13935
+ RET3936
+3937
+INIT_YMM avx23938
+cglobal psyRdoQuant_1p16, 7, 9, 163939
+ mov r4d, r4m3940
+ lea r0, [r0 + 2 * r4]3941
+ lea r4, [4 * r4]3942
+ lea r1, [r1 + 2 * r4]3943
+ movq xm0, [r2]3944
+ movq xm1, [r3]3945
+%if BIT_DEPTH == 123946
+ mov r5, [tab_nonpsyRdo12 + 16] ; scaleBits3947
+%elif BIT_DEPTH == 103948
+ mov r5, [tab_nonpsyRdo10 + 16]3949
+%elif BIT_DEPTH == 83950
+ mov r5, [tab_nonpsyRdo8 + 16 ]3951
+%else3952
+ %error Unsupported BIT_DEPTH!3953
+%endif3954
+ movq xm2, r53955
+ vpxor m4, m43956
+ vpxor m3, m33957
+ vpxor m13, m133958
+3959
+ vpmovsxwd m6, [r0]3960
+ vcvtdq2pd m9, xm63961
+ vfmadd213pd m9, m9, m33962
+ vcvtpd2dq xm8, m93963
+ vpmovsxdq m13, xm8 ; 32 bit int to 64 bit int3964
+ vpsllq m13, xm2 ;(signCoef * signCoef) << scaleBits 3965
+ paddq m4, m133966
+ movu [r1], m133967
+3968
+ vpmovsxwd m6, [r0 + mmsize]3969
+3970
+ vcvtdq2pd m9, xm63971
+ vfmadd213pd m9, m9, m33972
+ vcvtpd2dq xm8, m93973
+ vpmovsxdq m13, xm8 ; 32 bit int to 64 bit int3974
+ vpsllq m13, xm2 ;(signCoef * signCoef) << scaleBits 3975
+ paddq m4, m133976
+ movu [r1 + 4*mmsize], m133977
+3978
+ vpmovsxwd m6, [r0 + 2 * mmsize]3979
+ vcvtdq2pd m9, xm63980
+ vfmadd213pd m9, m9, m33981
+ vcvtpd2dq xm8, m93982
+ vpmovsxdq m13, xm8 ; 32 bit int to 64 bit int3983
+ vpsllq m13, xm2 ;(signCoef * signCoef) << scaleBits 3984
+ paddq m4, m133985
+ movu [r1 + 8*mmsize], m133986
+3987
+ vpmovsxwd m6, [r0 + 3 * mmsize]3988
+ vcvtdq2pd m9, xm63989
+ vfmadd213pd m9, m9, m33990
+ vcvtpd2dq xm8, m93991
+ vpmovsxdq m13, xm8 ; 32 bit int to 64 bit int3992
+ vpsllq m13, xm2 ;(signCoef * signCoef) << scaleBits 3993
+ paddq m4, m133994
+ movu [r1 + 12*mmsize], m133995
+3996
+ vextracti128 xm2, m4, 13997
+ paddq xm4, xm23998
+ punpckhqdq xm2, xm4, xm33999
+ paddq xm4, xm24000
+4001
+ paddq xm0, xm44002
+ paddq xm1, xm44003
+4004
+ movq [r2], xm04005
+ movq [r3], xm14006
+ RET4007
+4008
+INIT_YMM avx24009
+cglobal psyRdoQuant_1p32, 7, 9, 164010
+ mov r4d, r4m4011
+ lea r0, [r0 + 2 * r4]4012
+ lea r4, [4 * r4]4013
+ lea r1, [r1 + 2 * r4]4014
+ movq xm0, [r2]4015
+ movq xm1, [r3]4016
+%if BIT_DEPTH == 124017
+ mov r5, [tab_nonpsyRdo12 + 24] ; scaleBits4018
+%elif BIT_DEPTH == 104019
+ mov r5, [tab_nonpsyRdo10 + 24]4020
+%elif BIT_DEPTH == 84021
+ mov r5, [tab_nonpsyRdo8 + 24]4022
+%else4023
+ %error Unsupported BIT_DEPTH!4024
+%endif4025
+ movq xm2, r54026
+ vpxor m4, m44027
+ vpxor m3, m34028
+ vpxor m13, m134029
+4030
+4031
+ vpmovsxwd m6, [r0]4032
+ vcvtdq2pd m9, xm64033
+ vfmadd213pd m9, m9, m34034
+ vcvtpd2dq xm8, m94035
+ vpmovsxdq m13, xm8 ; 32 bit int to 64 bit int4036
+ vpsllq m13, xm2 ;(signCoef * signCoef) << scaleBits 4037
+ paddq m4, m134038
+ movu [r1], m134039
+4040
+ vpmovsxwd m6, [r0 + 2 * mmsize]4041
+ vcvtdq2pd m9, xm64042
+ vfmadd213pd m9, m9, m34043
+ vcvtpd2dq xm8, m94044
+ vpmovsxdq m13, xm8 ; 32 bit int to 64 bit int4045
+ vpsllq m13, xm2 ;(signCoef * signCoef) << scaleBits 4046
+ paddq m4, m134047
+ movu [r1 + 8 * mmsize], m134048
+4049
+ vpmovsxwd m6, [r0 + 4 * mmsize]4050
+ vcvtdq2pd m9, xm64051
+ vfmadd213pd m9, m9, m34052
+ vcvtpd2dq xm8, m94053
+ vpmovsxdq m13, xm8 ; 32 bit int to 64 bit int4054
+ vpsllq m13, xm2 ;(signCoef * signCoef) << scaleBits 4055
+ paddq m4, m134056
+ movu [r1 + 16 * mmsize], m134057
+4058
+ vpmovsxwd m6, [r0 + 6 * mmsize]4059
+ vcvtdq2pd m9, xm64060
+ vfmadd213pd m9, m9, m34061
+ vcvtpd2dq xm8, m94062
+ vpmovsxdq m13, xm8 ; 32 bit int to 64 bit int4063
+ vpsllq m13, xm2 ;(signCoef * signCoef) << scaleBits 4064
+ paddq m4, m134065
+ movu [r1 + 24 *mmsize], m134066
+4067
+ vextracti128 xm2, m4, 14068
+ paddq xm4, xm24069
+ punpckhqdq xm2, xm4, xm34070
+ paddq xm4, xm24071
+4072
+ paddq xm0, xm44073
+ paddq xm1, xm44074
+4075
+ movq [r2], xm04076
+ movq [r3], xm14077
+ RET4078
+4079
%endif4080
x265_2.7.tar.gz/source/common/x86/dct8.h -> x265_2.9.tar.gz/source/common/x86/dct8.h
Changed
26
1
2
FUNCDEF_TU_S2(void, idct, ssse3, const int16_t* src, int16_t* dst, intptr_t dstStride);3
FUNCDEF_TU_S2(void, idct, sse4, const int16_t* src, int16_t* dst, intptr_t dstStride);4
FUNCDEF_TU_S2(void, idct, avx2, const int16_t* src, int16_t* dst, intptr_t dstStride);5
+FUNCDEF_TU_S2(void, nonPsyRdoQuant, avx512, int16_t *m_resiDctCoeff, int64_t *costUncoded, int64_t *totalUncodedCost, int64_t *totalRdCost, uint32_t blkPos);6
+FUNCDEF_TU_S2(void, psyRdoQuant, avx512, int16_t* m_resiDctCoeff, int16_t* m_fencDctCoeff, int64_t* costUncoded, int64_t* totalUncodedCost, int64_t* totalRdCost, int64_t *psyScale, uint32_t blkPos);7
+FUNCDEF_TU_S2(void, nonPsyRdoQuant, avx2, int16_t *m_resiDctCoeff, int64_t *costUncoded, int64_t *totalUncodedCost, int64_t *totalRdCost, uint32_t blkPos);8
+FUNCDEF_TU_S2(void, psyRdoQuant_1p, avx2, int16_t* m_resiDctCoeff, int64_t* costUncoded, int64_t* totalUncodedCost, int64_t* totalRdCost, uint32_t blkPos);9
+FUNCDEF_TU_S2(void, psyRdoQuant_2p, avx2, int16_t* m_resiDctCoeff, int16_t* m_fencDctCoeff, int64_t* costUncoded, int64_t* totalUncodedCost, int64_t* totalRdCost, int64_t *psyScale, uint32_t blkPos);10
11
void PFX(dst4_ssse3)(const int16_t* src, int16_t* dst, intptr_t srcStride);12
void PFX(dst4_sse2)(const int16_t* src, int16_t* dst, intptr_t srcStride);13
14
void PFX(idst4_avx2)(const int16_t* src, int16_t* dst, intptr_t srcStride);15
void PFX(denoise_dct_sse4)(int16_t* dct, uint32_t* sum, const uint16_t* offset, int size);16
void PFX(denoise_dct_avx2)(int16_t* dct, uint32_t* sum, const uint16_t* offset, int size);17
-18
+void PFX(denoise_dct_avx512)(int16_t* dct, uint32_t* sum, const uint16_t* offset, int size);19
+void PFX(dct8_avx512)(const int16_t* src, int16_t* dst, intptr_t srcStride);20
+void PFX(idct8_avx512)(const int16_t* src, int16_t* dst, intptr_t dstStride);21
+void PFX(idct16_avx512)(const int16_t* src, int16_t* dst, intptr_t dstStride);22
+void PFX(idct32_avx512)(const int16_t* src, int16_t* dst, intptr_t dstStride);23
+void PFX(dct32_avx512)(const int16_t* src, int16_t* dst, intptr_t srcStride);24
+void PFX(dct16_avx512)(const int16_t* src, int16_t* dst, intptr_t srcStride);25
#endif // ifndef X265_DCT8_H26
x265_2.7.tar.gz/source/common/x86/h-ipfilter16.asm -> x265_2.9.tar.gz/source/common/x86/h-ipfilter16.asm
Changed
1589
1
2
3
h_pd_524800: times 8 dd 5248004
5
-tab_LumaCoeff: dw 0, 0, 0, 64, 0, 0, 0, 06
+h_tab_LumaCoeff: dw 0, 0, 0, 64, 0, 0, 0, 07
dw -1, 4, -10, 58, 17, -5, 1, 08
dw -1, 4, -11, 40, 40, -11, 4, -19
dw 0, 1, -5, 17, 58, -10, 4, -110
11
db 4, 5, 6, 7, 8, 9, 10, 11, 6, 7, 8, 9, 10, 11, 12, 13 12
13
const interp8_hpp_shuf_new, db 0, 1, 2, 3, 2, 3, 4, 5, 4, 5, 6, 7, 6, 7, 8, 914
- db 4, 5, 6, 7, 6, 7, 8, 9, 8, 9, 10, 11, 10, 11, 12, 13 15
- 16
+ db 4, 5, 6, 7, 6, 7, 8, 9, 8, 9, 10, 11, 10, 11, 12, 1317
+18
+ALIGN 6419
+interp8_hpp_shuf1_load_avx512: times 4 db 0, 1, 2, 3, 4, 5, 6, 7, 2, 3, 4, 5, 6, 7, 8, 920
+interp8_hpp_shuf2_load_avx512: times 4 db 4, 5, 6, 7, 8, 9, 10, 11, 6, 7, 8, 9, 10, 11, 12, 1321
+interp8_hpp_shuf1_store_avx512: times 4 db 0, 1, 4, 5, 2, 3, 6, 7, 8, 9, 12, 13, 10, 11, 14, 1522
+23
SECTION .text24
cextern pd_825
cextern pd_3226
27
add r3d, r3d28
29
%ifdef PIC30
- lea r6, [tab_LumaCoeff]31
+ lea r6, [h_tab_LumaCoeff]32
mova m0, [r6 + r4]33
%else34
- mova m0, [tab_LumaCoeff + r4]35
+ mova m0, [h_tab_LumaCoeff + r4]36
%endif37
38
%ifidn %3, pp39
40
;------------------------------------------------------------------------------------------------------------41
; void interp_8tap_horiz_pp_%1x%2(pixel *src, intptr_t srcStride, pixel *dst, intptr_t dstStride, int coeffIdx42
;------------------------------------------------------------------------------------------------------------43
- FILTER_HOR_LUMA_sse2 4, 4, pp44
+%if ARCH_X86_6445
+ FILTER_HOR_LUMA_sse2 4, 4, pp46
FILTER_HOR_LUMA_sse2 4, 8, pp47
FILTER_HOR_LUMA_sse2 4, 16, pp48
FILTER_HOR_LUMA_sse2 8, 4, pp49
50
FILTER_HOR_LUMA_sse2 64, 32, ps51
FILTER_HOR_LUMA_sse2 64, 48, ps52
FILTER_HOR_LUMA_sse2 64, 64, ps53
+%endif54
55
;-----------------------------------------------------------------------------56
; void interp_4tap_horiz_%3_%1x%2(pixel *src, intptr_t srcStride, pixel *dst, intptr_t dstStride, int coeffIdx)57
58
add r3, r359
60
%ifdef PIC61
- lea r6, [tab_LumaCoeff]62
+ lea r6, [h_tab_LumaCoeff]63
mova m0, [r6 + r4]64
%else65
- mova m0, [tab_LumaCoeff + r4]66
+ mova m0, [h_tab_LumaCoeff + r4]67
%endif68
69
%ifidn %3, pp70
71
shl r4d, 472
73
%ifdef PIC74
- lea r6, [tab_LumaCoeff]75
+ lea r6, [h_tab_LumaCoeff]76
mova m0, [r6 + r4]77
%else78
- mova m0, [tab_LumaCoeff + r4]79
+ mova m0, [h_tab_LumaCoeff + r4]80
%endif81
82
%ifidn %3, pp83
84
shl r4d, 485
86
%ifdef PIC87
- lea r6, [tab_LumaCoeff]88
+ lea r6, [h_tab_LumaCoeff]89
mova m0, [r6 + r4]90
%else91
- mova m0, [tab_LumaCoeff + r4]92
+ mova m0, [h_tab_LumaCoeff + r4]93
%endif94
%ifidn %3, pp95
mova m1, [INTERP_OFFSET_PP]96
97
shl r4d, 498
99
%ifdef PIC100
- lea r6, [tab_LumaCoeff]101
+ lea r6, [h_tab_LumaCoeff]102
mova m0, [r6 + r4]103
%else104
- mova m0, [tab_LumaCoeff + r4]105
+ mova m0, [h_tab_LumaCoeff + r4]106
%endif107
108
%ifidn %3, pp109
110
shl r4d, 4111
112
%ifdef PIC113
- lea r6, [tab_LumaCoeff]114
+ lea r6, [h_tab_LumaCoeff]115
mova m0, [r6 + r4]116
%else117
- mova m0, [tab_LumaCoeff + r4]118
+ mova m0, [h_tab_LumaCoeff + r4]119
%endif120
%ifidn %3, pp121
mova m1, [pd_32]122
123
mov r4d, r4m124
shl r4d, 4125
%ifdef PIC126
- lea r5, [tab_LumaCoeff]127
+ lea r5, [h_tab_LumaCoeff]128
vpbroadcastq m0, [r5 + r4]129
vpbroadcastq m1, [r5 + r4 + 8]130
%else131
- vpbroadcastq m0, [tab_LumaCoeff + r4]132
- vpbroadcastq m1, [tab_LumaCoeff + r4 + 8]133
+ vpbroadcastq m0, [h_tab_LumaCoeff + r4]134
+ vpbroadcastq m1, [h_tab_LumaCoeff + r4 + 8]135
%endif136
lea r6, [pw_pixel_max]137
mova m3, [interp8_hpp_shuf]138
139
;-------------------------------------------------------------------------------------------------------------140
; void interp_8tap_horiz_pp(pixel *src, intptr_t srcStride, pixel *dst, intptr_t dstStride, int coeffIdx141
;-------------------------------------------------------------------------------------------------------------142
-%macro FILTER_HOR_LUMA_W8 1143
+%macro PROCESS_IPFILTER_LUMA_PP_8x2_AVX2 0144
+ movu xm7, [r0]145
+ movu xm8, [r0 + 8]146
+ vinserti128 m7, m7, [r0 + r1], 1147
+ vinserti128 m8, m8, [r0 + r1 + 8], 1148
+ pshufb m10, m7, m14149
+ pshufb m7, m13150
+ pshufb m11, m8, m14151
+ pshufb m8, m13152
+153
+ pmaddwd m7, m0154
+ pmaddwd m10, m1155
+ paddd m7, m10156
+ pmaddwd m10, m11, m3157
+ pmaddwd m9, m8, m2158
+ paddd m10, m9159
+ paddd m7, m10160
+ paddd m7, m4161
+ psrad m7, INTERP_SHIFT_PP162
+163
+ movu xm9, [r0 + 16]164
+ vinserti128 m9, m9, [r0 + r1 + 16], 1165
+ pshufb m10, m9, m14166
+ pshufb m9, m13167
+ pmaddwd m8, m0168
+ pmaddwd m11, m1169
+ paddd m8, m11170
+ pmaddwd m10, m3171
+ pmaddwd m9, m2172
+ paddd m9, m10173
+ paddd m8, m9174
+ paddd m8, m4175
+ psrad m8, INTERP_SHIFT_PP176
+177
+ packusdw m7, m8178
+ pshufb m7, m12179
+ CLIPW m7, m5, m6180
+ movu [r2], xm7181
+ vextracti128 [r2 + r3], m7, 1182
+%endmacro183
+184
+%macro IPFILTER_LUMA_AVX2_8xN 1185
INIT_YMM avx2186
-cglobal interp_8tap_horiz_pp_8x%1, 4,6,8187
- add r1d, r1d188
- add r3d, r3d189
- sub r0, 6190
- mov r4d, r4m191
- shl r4d, 4192
+cglobal interp_8tap_horiz_pp_8x%1, 5,6,15193
+ shl r1d, 1194
+ shl r3d, 1195
+ sub r0, 6196
+ mov r4d, r4m197
+ shl r4d, 4198
+199
%ifdef PIC200
- lea r5, [tab_LumaCoeff]201
- vpbroadcastq m0, [r5 + r4]202
- vpbroadcastq m1, [r5 + r4 + 8]203
+ lea r5, [h_tab_LumaCoeff]204
+ vpbroadcastd m0, [r5 + r4]205
+ vpbroadcastd m1, [r5 + r4 + 4]206
+ vpbroadcastd m2, [r5 + r4 + 8]207
+ vpbroadcastd m3, [r5 + r4 + 12]208
%else209
- vpbroadcastq m0, [tab_LumaCoeff + r4]210
- vpbroadcastq m1, [h_ab_LumaCoeff + r4 + 8]211
-%endif212
- mova m3, [interp8_hpp_shuf]213
- mova m7, [pd_32]214
- pxor m2, m2215
-216
- ; register map217
- ; m0 , m1 interpolate coeff218
-219
- mov r4d, %1/2220
-221
-.loop:222
- vbroadcasti128 m4, [r0]223
- vbroadcasti128 m5, [r0 + 8]224
- pshufb m4, m3225
- pshufb m5, m3226
-227
- pmaddwd m4, m0228
- pmaddwd m5, m1229
- paddd m4, m5230
-231
- vbroadcasti128 m5, [r0 + 8]232
- vbroadcasti128 m6, [r0 + 16]233
- pshufb m5, m3234
- pshufb m6, m3235
-236
- pmaddwd m5, m0237
- pmaddwd m6, m1238
- paddd m5, m6239
-240
- phaddd m4, m5241
- vpermq m4, m4, q3120242
- paddd m4, m7243
- psrad m4, INTERP_SHIFT_PP244
-245
- packusdw m4, m4246
- vpermq m4, m4, q2020247
- CLIPW m4, m2, [pw_pixel_max]248
- movu [r2], xm4249
-250
- vbroadcasti128 m4, [r0 + r1]251
- vbroadcasti128 m5, [r0 + r1 + 8]252
- pshufb m4, m3253
- pshufb m5, m3254
-255
- pmaddwd m4, m0256
- pmaddwd m5, m1257
- paddd m4, m5258
-259
- vbroadcasti128 m5, [r0 + r1 + 8]260
- vbroadcasti128 m6, [r0 + r1 + 16]261
- pshufb m5, m3262
- pshufb m6, m3263
-264
- pmaddwd m5, m0265
- pmaddwd m6, m1266
- paddd m5, m6267
-268
- phaddd m4, m5269
- vpermq m4, m4, q3120270
- paddd m4, m7271
- psrad m4, INTERP_SHIFT_PP272
-273
- packusdw m4, m4274
- vpermq m4, m4, q2020275
- CLIPW m4, m2, [pw_pixel_max]276
- movu [r2 + r3], xm4277
-278
- lea r2, [r2 + 2 * r3]279
- lea r0, [r0 + 2 * r1]280
- dec r4d281
- jnz .loop282
+ vpbroadcastd m0, [h_tab_LumaCoeff + r4]283
+ vpbroadcastd m1, [h_tab_LumaCoeff + r4 + 4]284
+ vpbroadcastd m2, [h_tab_LumaCoeff + r4 + 8]285
+ vpbroadcastd m3, [h_tab_LumaCoeff + r4 + 12]286
+%endif287
+ mova m13, [interp8_hpp_shuf1_load_avx512]288
+ mova m14, [interp8_hpp_shuf2_load_avx512]289
+ mova m12, [interp8_hpp_shuf1_store_avx512]290
+ mova m4, [pd_32]291
+ pxor m5, m5292
+ mova m6, [pw_pixel_max]293
+294
+%rep %1/2 - 1295
+ PROCESS_IPFILTER_LUMA_PP_8x2_AVX2296
+ lea r0, [r0 + 2 * r1]297
+ lea r2, [r2 + 2 * r3]298
+%endrep299
+ PROCESS_IPFILTER_LUMA_PP_8x2_AVX2300
RET301
%endmacro302
-FILTER_HOR_LUMA_W8 4303
-FILTER_HOR_LUMA_W8 8304
-FILTER_HOR_LUMA_W8 16305
-FILTER_HOR_LUMA_W8 32306
+307
+%if ARCH_X86_64308
+ IPFILTER_LUMA_AVX2_8xN 4309
+ IPFILTER_LUMA_AVX2_8xN 8310
+ IPFILTER_LUMA_AVX2_8xN 16311
+ IPFILTER_LUMA_AVX2_8xN 32312
+%endif313
314
;-------------------------------------------------------------------------------------------------------------315
; void interp_8tap_horiz_pp(pixel *src, intptr_t srcStride, pixel *dst, intptr_t dstStride, int coeffIdx316
;-------------------------------------------------------------------------------------------------------------317
-%macro FILTER_HOR_LUMA_W16 1318
+%macro PROCESS_IPFILTER_LUMA_PP_16x1_AVX2 0319
+ movu m7, [r0]320
+ movu m8, [r0 + 8]321
+322
+ pshufb m10, m7, m14323
+ pshufb m7, m13324
+ pshufb m11, m8, m14325
+ pshufb m8, m13326
+327
+ pmaddwd m7, m0328
+ pmaddwd m10, m1329
+ paddd m7, m10330
+ pmaddwd m10, m11, m3331
+ pmaddwd m9, m8, m2332
+ paddd m10, m9333
+ paddd m7, m10334
+ paddd m7, m4335
+ psrad m7, INTERP_SHIFT_PP336
+337
+ movu m9, [r0 + 16]338
+ pshufb m10, m9, m14339
+ pshufb m9, m13340
+ pmaddwd m8, m0341
+ pmaddwd m11, m1342
+ paddd m8, m11343
+ pmaddwd m10, m3344
+ pmaddwd m9, m2345
+ paddd m9, m10346
+ paddd m8, m9347
+ paddd m8, m4348
+ psrad m8, INTERP_SHIFT_PP349
+350
+ packusdw m7, m8351
+ pshufb m7, m12352
+ CLIPW m7, m5, m6353
+ movu [r2], m7354
+%endmacro355
+356
+%macro IPFILTER_LUMA_AVX2_16xN 1357
INIT_YMM avx2358
-cglobal interp_8tap_horiz_pp_16x%1, 4,6,8359
- add r1d, r1d360
- add r3d, r3d361
- sub r0, 6362
- mov r4d, r4m363
- shl r4d, 4364
+cglobal interp_8tap_horiz_pp_16x%1, 5,6,15365
+ shl r1d, 1366
+ shl r3d, 1367
+ sub r0, 6368
+ mov r4d, r4m369
+ shl r4d, 4370
+371
%ifdef PIC372
- lea r5, [tab_LumaCoeff]373
- vpbroadcastq m0, [r5 + r4]374
- vpbroadcastq m1, [r5 + r4 + 8]375
+ lea r5, [h_tab_LumaCoeff]376
+ vpbroadcastd m0, [r5 + r4]377
+ vpbroadcastd m1, [r5 + r4 + 4]378
+ vpbroadcastd m2, [r5 + r4 + 8]379
+ vpbroadcastd m3, [r5 + r4 + 12]380
%else381
- vpbroadcastq m0, [tab_LumaCoeff + r4]382
- vpbroadcastq m1, [tab_LumaCoeff + r4 + 8]383
-%endif384
- mova m3, [interp8_hpp_shuf]385
- mova m7, [pd_32]386
- pxor m2, m2387
-388
- ; register map389
- ; m0 , m1 interpolate coeff390
-391
- mov r4d, %1392
-393
-.loop:394
- vbroadcasti128 m4, [r0]395
- vbroadcasti128 m5, [r0 + 8]396
- pshufb m4, m3397
- pshufb m5, m3398
-399
- pmaddwd m4, m0400
- pmaddwd m5, m1401
- paddd m4, m5402
-403
- vbroadcasti128 m5, [r0 + 8]404
- vbroadcasti128 m6, [r0 + 16]405
- pshufb m5, m3406
- pshufb m6, m3407
-408
- pmaddwd m5, m0409
- pmaddwd m6, m1410
- paddd m5, m6411
-412
- phaddd m4, m5413
- vpermq m4, m4, q3120414
- paddd m4, m7415
- psrad m4, INTERP_SHIFT_PP416
-417
- packusdw m4, m4418
- vpermq m4, m4, q2020419
- CLIPW m4, m2, [pw_pixel_max]420
- movu [r2], xm4421
-422
- vbroadcasti128 m4, [r0 + 16]423
- vbroadcasti128 m5, [r0 + 24]424
- pshufb m4, m3425
- pshufb m5, m3426
-427
- pmaddwd m4, m0428
- pmaddwd m5, m1429
- paddd m4, m5430
-431
- vbroadcasti128 m5, [r0 + 24]432
- vbroadcasti128 m6, [r0 + 32]433
- pshufb m5, m3434
- pshufb m6, m3435
-436
- pmaddwd m5, m0437
- pmaddwd m6, m1438
- paddd m5, m6439
-440
- phaddd m4, m5441
- vpermq m4, m4, q3120442
- paddd m4, m7443
- psrad m4, INTERP_SHIFT_PP444
-445
- packusdw m4, m4446
- vpermq m4, m4, q2020447
- CLIPW m4, m2, [pw_pixel_max]448
- movu [r2 + 16], xm4449
-450
- add r2, r3451
- add r0, r1452
- dec r4d453
- jnz .loop454
+ vpbroadcastd m0, [h_tab_LumaCoeff + r4]455
+ vpbroadcastd m1, [h_tab_LumaCoeff + r4 + 4]456
+ vpbroadcastd m2, [h_tab_LumaCoeff + r4 + 8]457
+ vpbroadcastd m3, [h_tab_LumaCoeff + r4 + 12]458
+%endif459
+ mova m13, [interp8_hpp_shuf1_load_avx512]460
+ mova m14, [interp8_hpp_shuf2_load_avx512]461
+ mova m12, [interp8_hpp_shuf1_store_avx512]462
+ mova m4, [pd_32]463
+ pxor m5, m5464
+ mova m6, [pw_pixel_max]465
+466
+%rep %1 - 1467
+ PROCESS_IPFILTER_LUMA_PP_16x1_AVX2468
+ lea r0, [r0 + r1]469
+ lea r2, [r2 + r3]470
+%endrep471
+ PROCESS_IPFILTER_LUMA_PP_16x1_AVX2472
RET473
%endmacro474
-FILTER_HOR_LUMA_W16 4475
-FILTER_HOR_LUMA_W16 8476
-FILTER_HOR_LUMA_W16 12477
-FILTER_HOR_LUMA_W16 16478
-FILTER_HOR_LUMA_W16 32479
-FILTER_HOR_LUMA_W16 64480
+481
+%if ARCH_X86_64482
+ IPFILTER_LUMA_AVX2_16xN 4483
+ IPFILTER_LUMA_AVX2_16xN 8484
+ IPFILTER_LUMA_AVX2_16xN 12485
+ IPFILTER_LUMA_AVX2_16xN 16486
+ IPFILTER_LUMA_AVX2_16xN 32487
+ IPFILTER_LUMA_AVX2_16xN 64488
+%endif489
490
;-------------------------------------------------------------------------------------------------------------491
; void interp_8tap_horiz_pp(pixel *src, intptr_t srcStride, pixel *dst, intptr_t dstStride, int coeffIdx492
;-------------------------------------------------------------------------------------------------------------493
-%macro FILTER_HOR_LUMA_W32 2494
+%macro PROCESS_IPFILTER_LUMA_PP_32x1_AVX2 0495
+ PROCESS_IPFILTER_LUMA_PP_16x1_AVX2496
+497
+ movu m7, [r0 + mmsize]498
+ movu m8, [r0 + 8 + mmsize]499
+500
+ pshufb m10, m7, m14501
+ pshufb m7, m13502
+ pshufb m11, m8, m14503
+ pshufb m8, m13504
+505
+ pmaddwd m7, m0506
+ pmaddwd m10, m1507
+ paddd m7, m10508
+ pmaddwd m10, m11, m3509
+ pmaddwd m9, m8, m2510
+ paddd m10, m9511
+ paddd m7, m10512
+ paddd m7, m4513
+ psrad m7, INTERP_SHIFT_PP514
+515
+ movu m9, [r0 + 16 + mmsize]516
+ pshufb m10, m9, m14517
+ pshufb m9, m13518
+ pmaddwd m8, m0519
+ pmaddwd m11, m1520
+ paddd m8, m11521
+ pmaddwd m10, m3522
+ pmaddwd m9, m2523
+ paddd m9, m10524
+ paddd m8, m9525
+ paddd m8, m4526
+ psrad m8, INTERP_SHIFT_PP527
+528
+ packusdw m7, m8529
+ pshufb m7, m12530
+ CLIPW m7, m5, m6531
+ movu [r2 + mmsize], m7532
+%endmacro533
+534
+%macro IPFILTER_LUMA_AVX2_32xN 1535
INIT_YMM avx2536
-cglobal interp_8tap_horiz_pp_%1x%2, 4,6,8537
- add r1d, r1d538
- add r3d, r3d539
- sub r0, 6540
- mov r4d, r4m541
- shl r4d, 4542
+cglobal interp_8tap_horiz_pp_32x%1, 5,6,15543
+ shl r1d, 1544
+ shl r3d, 1545
+ sub r0, 6546
+ mov r4d, r4m547
+ shl r4d, 4548
+549
%ifdef PIC550
- lea r5, [tab_LumaCoeff]551
- vpbroadcastq m0, [r5 + r4]552
- vpbroadcastq m1, [r5 + r4 + 8]553
+ lea r5, [h_tab_LumaCoeff]554
+ vpbroadcastd m0, [r5 + r4]555
+ vpbroadcastd m1, [r5 + r4 + 4]556
+ vpbroadcastd m2, [r5 + r4 + 8]557
+ vpbroadcastd m3, [r5 + r4 + 12]558
%else559
- vpbroadcastq m0, [tab_LumaCoeff + r4]560
- vpbroadcastq m1, [tab_LumaCoeff + r4 + 8]561
-%endif562
- mova m3, [interp8_hpp_shuf]563
- mova m7, [pd_32]564
- pxor m2, m2565
-566
- ; register map567
- ; m0 , m1 interpolate coeff568
-569
- mov r4d, %2570
-571
-.loop:572
-%assign x 0573
-%rep %1/16574
- vbroadcasti128 m4, [r0 + x]575
- vbroadcasti128 m5, [r0 + 8 + x]576
- pshufb m4, m3577
- pshufb m5, m3578
-579
- pmaddwd m4, m0580
- pmaddwd m5, m1581
- paddd m4, m5582
-583
- vbroadcasti128 m5, [r0 + 8 + x]584
- vbroadcasti128 m6, [r0 + 16 + x]585
- pshufb m5, m3586
- pshufb m6, m3587
-588
- pmaddwd m5, m0589
- pmaddwd m6, m1590
- paddd m5, m6591
-592
- phaddd m4, m5593
- vpermq m4, m4, q3120594
- paddd m4, m7595
- psrad m4, INTERP_SHIFT_PP596
-597
- packusdw m4, m4598
- vpermq m4, m4, q2020599
- CLIPW m4, m2, [pw_pixel_max]600
- movu [r2 + x], xm4601
-602
- vbroadcasti128 m4, [r0 + 16 + x]603
- vbroadcasti128 m5, [r0 + 24 + x]604
- pshufb m4, m3605
- pshufb m5, m3606
-607
- pmaddwd m4, m0608
- pmaddwd m5, m1609
- paddd m4, m5610
-611
- vbroadcasti128 m5, [r0 + 24 + x]612
- vbroadcasti128 m6, [r0 + 32 + x]613
- pshufb m5, m3614
- pshufb m6, m3615
-616
- pmaddwd m5, m0617
- pmaddwd m6, m1618
- paddd m5, m6619
-620
- phaddd m4, m5621
- vpermq m4, m4, q3120622
- paddd m4, m7623
- psrad m4, INTERP_SHIFT_PP624
-625
- packusdw m4, m4626
- vpermq m4, m4, q2020627
- CLIPW m4, m2, [pw_pixel_max]628
- movu [r2 + 16 + x], xm4629
+ vpbroadcastd m0, [h_tab_LumaCoeff + r4]630
+ vpbroadcastd m1, [h_tab_LumaCoeff + r4 + 4]631
+ vpbroadcastd m2, [h_tab_LumaCoeff + r4 + 8]632
+ vpbroadcastd m3, [h_tab_LumaCoeff + r4 + 12]633
+%endif634
+ mova m13, [interp8_hpp_shuf1_load_avx512]635
+ mova m14, [interp8_hpp_shuf2_load_avx512]636
+ mova m12, [interp8_hpp_shuf1_store_avx512]637
+ mova m4, [pd_32]638
+ pxor m5, m5639
+ mova m6, [pw_pixel_max]640
+641
+%rep %1 - 1642
+ PROCESS_IPFILTER_LUMA_PP_32x1_AVX2643
+ lea r0, [r0 + r1]644
+ lea r2, [r2 + r3]645
+%endrep646
+ PROCESS_IPFILTER_LUMA_PP_32x1_AVX2647
+ RET648
+%endmacro649
650
+%if ARCH_X86_64651
+ IPFILTER_LUMA_AVX2_32xN 8652
+ IPFILTER_LUMA_AVX2_32xN 16653
+ IPFILTER_LUMA_AVX2_32xN 24654
+ IPFILTER_LUMA_AVX2_32xN 32655
+ IPFILTER_LUMA_AVX2_32xN 64656
+%endif657
+658
+%macro PROCESS_IPFILTER_LUMA_PP_64x1_AVX2 0659
+ PROCESS_IPFILTER_LUMA_PP_16x1_AVX2660
+%assign x 32661
+%rep 3662
+ movu m7, [r0 + x]663
+ movu m8, [r0 + 8 + x]664
+665
+ pshufb m10, m7, m14666
+ pshufb m7, m13667
+ pshufb m11, m8, m14668
+ pshufb m8, m13669
+670
+ pmaddwd m7, m0671
+ pmaddwd m10, m1672
+ paddd m7, m10673
+ pmaddwd m10, m11, m3674
+ pmaddwd m9, m8, m2675
+ paddd m10, m9676
+ paddd m7, m10677
+ paddd m7, m4678
+ psrad m7, INTERP_SHIFT_PP679
+680
+ movu m9, [r0 + 16 + x]681
+ pshufb m10, m9, m14682
+ pshufb m9, m13683
+ pmaddwd m8, m0684
+ pmaddwd m11, m1685
+ paddd m8, m11686
+ pmaddwd m10, m3687
+ pmaddwd m9, m2688
+ paddd m9, m10689
+ paddd m8, m9690
+ paddd m8, m4691
+ psrad m8, INTERP_SHIFT_PP692
+693
+ packusdw m7, m8694
+ pshufb m7, m12695
+ CLIPW m7, m5, m6696
+ movu [r2 + x], m7697
%assign x x+32698
%endrep699
+%endmacro700
701
- add r2, r3702
- add r0, r1703
- dec r4d704
- jnz .loop705
+%macro IPFILTER_LUMA_AVX2_64xN 1706
+INIT_YMM avx2707
+cglobal interp_8tap_horiz_pp_64x%1, 5,6,15708
+ shl r1d, 1709
+ shl r3d, 1710
+ sub r0, 6711
+ mov r4d, r4m712
+ shl r4d, 4713
+714
+%ifdef PIC715
+ lea r5, [h_tab_LumaCoeff]716
+ vpbroadcastd m0, [r5 + r4]717
+ vpbroadcastd m1, [r5 + r4 + 4]718
+ vpbroadcastd m2, [r5 + r4 + 8]719
+ vpbroadcastd m3, [r5 + r4 + 12]720
+%else721
+ vpbroadcastd m0, [h_tab_LumaCoeff + r4]722
+ vpbroadcastd m1, [h_tab_LumaCoeff + r4 + 4]723
+ vpbroadcastd m2, [h_tab_LumaCoeff + r4 + 8]724
+ vpbroadcastd m3, [h_tab_LumaCoeff + r4 + 12]725
+%endif726
+ mova m13, [interp8_hpp_shuf1_load_avx512]727
+ mova m14, [interp8_hpp_shuf2_load_avx512]728
+ mova m12, [interp8_hpp_shuf1_store_avx512]729
+ mova m4, [pd_32]730
+ pxor m5, m5731
+ mova m6, [pw_pixel_max]732
+733
+%rep %1 - 1734
+ PROCESS_IPFILTER_LUMA_PP_64x1_AVX2735
+ lea r0, [r0 + r1]736
+ lea r2, [r2 + r3]737
+%endrep738
+ PROCESS_IPFILTER_LUMA_PP_64x1_AVX2739
RET740
%endmacro741
-FILTER_HOR_LUMA_W32 32, 8742
-FILTER_HOR_LUMA_W32 32, 16743
-FILTER_HOR_LUMA_W32 32, 24744
-FILTER_HOR_LUMA_W32 32, 32745
-FILTER_HOR_LUMA_W32 32, 64746
-FILTER_HOR_LUMA_W32 64, 16747
-FILTER_HOR_LUMA_W32 64, 32748
-FILTER_HOR_LUMA_W32 64, 48749
-FILTER_HOR_LUMA_W32 64, 64750
+751
+%if ARCH_X86_64752
+ IPFILTER_LUMA_AVX2_64xN 16753
+ IPFILTER_LUMA_AVX2_64xN 32754
+ IPFILTER_LUMA_AVX2_64xN 48755
+ IPFILTER_LUMA_AVX2_64xN 64756
+%endif757
758
;-------------------------------------------------------------------------------------------------------------759
; void interp_8tap_horiz_pp(pixel *src, intptr_t srcStride, pixel *dst, intptr_t dstStride, int coeffIdx760
761
mov r4d, r4m762
shl r4d, 4763
%ifdef PIC764
- lea r5, [tab_LumaCoeff]765
+ lea r5, [h_tab_LumaCoeff]766
vpbroadcastq m0, [r5 + r4]767
vpbroadcastq m1, [r5 + r4 + 8]768
%else769
- vpbroadcastq m0, [tab_LumaCoeff + r4]770
- vpbroadcastq m1, [tab_LumaCoeff + r4 + 8]771
+ vpbroadcastq m0, [h_tab_LumaCoeff + r4]772
+ vpbroadcastq m1, [h_tab_LumaCoeff + r4 + 8]773
%endif774
mova m3, [interp8_hpp_shuf]775
mova m7, [pd_32]776
777
mov r4d, r4m778
shl r4d, 4779
%ifdef PIC780
- lea r5, [tab_LumaCoeff]781
+ lea r5, [h_tab_LumaCoeff]782
vpbroadcastq m0, [r5 + r4]783
vpbroadcastq m1, [r5 + r4 + 8]784
%else785
- vpbroadcastq m0, [tab_LumaCoeff + r4]786
- vpbroadcastq m1, [tab_LumaCoeff + r4 + 8]787
+ vpbroadcastq m0, [h_tab_LumaCoeff + r4]788
+ vpbroadcastq m1, [h_tab_LumaCoeff + r4 + 8]789
%endif790
mova m3, [interp8_hpp_shuf]791
mova m7, [pd_32]792
793
;-------------------------------------------------------------------------------------------------------------794
; void interp_8tap_horiz_pp(pixel *src, intptr_t srcStride, pixel *dst, intptr_t dstStride, int coeffIdx795
;-------------------------------------------------------------------------------------------------------------796
+%macro PROCESS_IPFILTER_LUMA_PP_48x1_AVX2 0797
+ PROCESS_IPFILTER_LUMA_PP_32x1_AVX2798
+799
+ movu m7, [r0 + 2 * mmsize]800
+ movu m8, [r0 + 8 + 2 * mmsize]801
+802
+ pshufb m10, m7, m14803
+ pshufb m7, m13804
+ pshufb m11, m8, m14805
+ pshufb m8, m13806
+807
+ pmaddwd m7, m0808
+ pmaddwd m10, m1809
+ paddd m7, m10810
+ pmaddwd m10, m11, m3811
+ pmaddwd m9, m8, m2812
+ paddd m10, m9813
+ paddd m7, m10814
+ paddd m7, m4815
+ psrad m7, INTERP_SHIFT_PP816
+817
+ movu m9, [r0 + 16 + 2 * mmsize]818
+ pshufb m10, m9, m14819
+ pshufb m9, m13820
+ pmaddwd m8, m0821
+ pmaddwd m11, m1822
+ paddd m8, m11823
+ pmaddwd m10, m3824
+ pmaddwd m9, m2825
+ paddd m9, m10826
+ paddd m8, m9827
+ paddd m8, m4828
+ psrad m8, INTERP_SHIFT_PP829
+830
+ packusdw m7, m8831
+ pshufb m7, m12832
+ CLIPW m7, m5, m6833
+ movu [r2 + 2 * mmsize], m7834
+%endmacro835
+836
+%if ARCH_X86_64837
INIT_YMM avx2838
-cglobal interp_8tap_horiz_pp_48x64, 4,6,8839
- add r1d, r1d840
- add r3d, r3d841
- sub r0, 6842
- mov r4d, r4m843
- shl r4d, 4844
+cglobal interp_8tap_horiz_pp_48x64, 5,6,15845
+ shl r1d, 1846
+ shl r3d, 1847
+ sub r0, 6848
+ mov r4d, r4m849
+ shl r4d, 4850
+851
%ifdef PIC852
- lea r5, [tab_LumaCoeff]853
- vpbroadcastq m0, [r5 + r4]854
- vpbroadcastq m1, [r5 + r4 + 8]855
+ lea r5, [h_tab_LumaCoeff]856
+ vpbroadcastd m0, [r5 + r4]857
+ vpbroadcastd m1, [r5 + r4 + 4]858
+ vpbroadcastd m2, [r5 + r4 + 8]859
+ vpbroadcastd m3, [r5 + r4 + 12]860
%else861
- vpbroadcastq m0, [tab_LumaCoeff + r4]862
- vpbroadcastq m1, [tab_LumaCoeff + r4 + 8]863
-%endif864
- mova m3, [interp8_hpp_shuf]865
- mova m7, [pd_32]866
- pxor m2, m2867
-868
- ; register map869
- ; m0 , m1 interpolate coeff870
-871
- mov r4d, 64872
-873
-.loop:874
-%assign x 0875
-%rep 2876
- vbroadcasti128 m4, [r0 + x]877
- vbroadcasti128 m5, [r0 + 8 + x]878
- pshufb m4, m3879
- pshufb m5, m3880
-881
- pmaddwd m4, m0882
- pmaddwd m5, m1883
- paddd m4, m5884
-885
- vbroadcasti128 m5, [r0 + 8 + x]886
- vbroadcasti128 m6, [r0 + 16 + x]887
- pshufb m5, m3888
- pshufb m6, m3889
-890
- pmaddwd m5, m0891
- pmaddwd m6, m1892
- paddd m5, m6893
-894
- phaddd m4, m5895
- vpermq m4, m4, q3120896
- paddd m4, m7897
- psrad m4, INTERP_SHIFT_PP898
-899
- packusdw m4, m4900
- vpermq m4, m4, q2020901
- CLIPW m4, m2, [pw_pixel_max]902
- movu [r2 + x], xm4903
-904
- vbroadcasti128 m4, [r0 + 16 + x]905
- vbroadcasti128 m5, [r0 + 24 + x]906
- pshufb m4, m3907
- pshufb m5, m3908
-909
- pmaddwd m4, m0910
- pmaddwd m5, m1911
- paddd m4, m5912
-913
- vbroadcasti128 m5, [r0 + 24 + x]914
- vbroadcasti128 m6, [r0 + 32 + x]915
- pshufb m5, m3916
- pshufb m6, m3917
-918
- pmaddwd m5, m0919
- pmaddwd m6, m1920
- paddd m5, m6921
-922
- phaddd m4, m5923
- vpermq m4, m4, q3120924
- paddd m4, m7925
- psrad m4, INTERP_SHIFT_PP926
-927
- packusdw m4, m4928
- vpermq m4, m4, q2020929
- CLIPW m4, m2, [pw_pixel_max]930
- movu [r2 + 16 + x], xm4931
-932
- vbroadcasti128 m4, [r0 + 32 + x]933
- vbroadcasti128 m5, [r0 + 40 + x]934
- pshufb m4, m3935
- pshufb m5, m3936
-937
- pmaddwd m4, m0938
- pmaddwd m5, m1939
- paddd m4, m5940
-941
- vbroadcasti128 m5, [r0 + 40 + x]942
- vbroadcasti128 m6, [r0 + 48 + x]943
- pshufb m5, m3944
- pshufb m6, m3945
-946
- pmaddwd m5, m0947
- pmaddwd m6, m1948
- paddd m5, m6949
-950
- phaddd m4, m5951
- vpermq m4, m4, q3120952
- paddd m4, m7953
- psrad m4, INTERP_SHIFT_PP954
-955
- packusdw m4, m4956
- vpermq m4, m4, q2020957
- CLIPW m4, m2, [pw_pixel_max]958
- movu [r2 + 32 + x], xm4959
-960
-%assign x x+48961
+ vpbroadcastd m0, [h_tab_LumaCoeff + r4]962
+ vpbroadcastd m1, [h_tab_LumaCoeff + r4 + 4]963
+ vpbroadcastd m2, [h_tab_LumaCoeff + r4 + 8]964
+ vpbroadcastd m3, [h_tab_LumaCoeff + r4 + 12]965
+%endif966
+ mova m13, [interp8_hpp_shuf1_load_avx512]967
+ mova m14, [interp8_hpp_shuf2_load_avx512]968
+ mova m12, [interp8_hpp_shuf1_store_avx512]969
+ mova m4, [pd_32]970
+ pxor m5, m5971
+ mova m6, [pw_pixel_max]972
+973
+%rep 63974
+ PROCESS_IPFILTER_LUMA_PP_48x1_AVX2975
+ lea r0, [r0 + r1]976
+ lea r2, [r2 + r3]977
%endrep978
-979
- add r2, r3980
- add r0, r1981
- dec r4d982
- jnz .loop983
+ PROCESS_IPFILTER_LUMA_PP_48x1_AVX2984
RET985
+%endif986
987
;-----------------------------------------------------------------------------------------------------------------------------988
;void interp_horiz_ps_c(const pixel* src, intptr_t srcStride, int16_t* dst, intptr_t dstStride, int coeffIdx, int isRowExt)989
990
add r3d, r3d991
992
%ifdef PIC993
- lea r6, [tab_LumaCoeff]994
+ lea r6, [h_tab_LumaCoeff]995
lea r4, [r4 * 8]996
vbroadcasti128 m0, [r6 + r4 * 2]997
%else998
lea r4, [r4 * 8]999
- vbroadcasti128 m0, [tab_LumaCoeff + r4 * 2]1000
+ vbroadcasti128 m0, [h_tab_LumaCoeff + r4 * 2]1001
%endif1002
1003
vbroadcasti128 m2, [INTERP_OFFSET_PS]1004
1005
IPFILTER_LUMA_PS_4xN_AVX2 81006
IPFILTER_LUMA_PS_4xN_AVX2 161007
1008
+ %macro PROCESS_IPFILTER_LUMA_PS_8x1_AVX2 11009
+1010
+ %assign x 01011
+ %rep %1/81012
+ vbroadcasti128 m4, [r0 + x]1013
+ vbroadcasti128 m5, [r0 + 8+ x]1014
+ pshufb m4, m31015
+ pshufb m7, m5, m31016
+ pmaddwd m4, m01017
+ pmaddwd m7, m11018
+ paddd m4, m71019
+1020
+ vbroadcasti128 m6, [r0 + 16 + x]1021
+ pshufb m5, m31022
+ pshufb m6, m31023
+ pmaddwd m5, m01024
+ pmaddwd m6, m11025
+ paddd m5, m61026
+1027
+ phaddd m4, m51028
+ vpermq m4, m4, q31201029
+ paddd m4, m21030
+ vextracti128 xm5,m4, 11031
+ psrad xm4, INTERP_SHIFT_PS1032
+ psrad xm5, INTERP_SHIFT_PS1033
+ packssdw xm4, xm51034
+ movu [r2 + x], xm41035
+ %assign x x+161036
+ %endrep1037
+ %endmacro1038
+1039
%macro IPFILTER_LUMA_PS_8xN_AVX2 11040
INIT_YMM avx21041
%if ARCH_X86_64 == 11042
cglobal interp_8tap_horiz_ps_8x%1, 4, 6, 81043
- add r1d, r1d1044
- add r3d, r3d1045
+ shl r1d, 11046
+ shl r3d, 11047
mov r4d, r4m1048
mov r5d, r5m1049
shl r4d, 41050
%ifdef PIC1051
- lea r6, [tab_LumaCoeff]1052
+ lea r6, [h_tab_LumaCoeff]1053
vpbroadcastq m0, [r6 + r4]1054
vpbroadcastq m1, [r6 + r4 + 8]1055
%else1056
- vpbroadcastq m0, [tab_LumaCoeff + r4]1057
- vpbroadcastq m1, [tab_LumaCoeff + r4 + 8]1058
+ vpbroadcastq m0, [h_tab_LumaCoeff + r4]1059
+ vpbroadcastq m1, [h_tab_LumaCoeff + r4 + 8]1060
%endif1061
mova m3, [interp8_hpp_shuf]1062
vbroadcasti128 m2, [INTERP_OFFSET_PS]1063
1064
add r4d, 71065
1066
.loop0:1067
- vbroadcasti128 m4, [r0]1068
- vbroadcasti128 m5, [r0 + 8]1069
- pshufb m4, m31070
- pshufb m7, m5, m31071
- pmaddwd m4, m01072
- pmaddwd m7, m11073
- paddd m4, m71074
-1075
- vbroadcasti128 m6, [r0 + 16]1076
- pshufb m5, m31077
- pshufb m6, m31078
- pmaddwd m5, m01079
- pmaddwd m6, m11080
- paddd m5, m61081
-1082
- phaddd m4, m51083
- vpermq m4, m4, q31201084
- paddd m4, m21085
- vextracti128 xm5,m4, 11086
- psrad xm4, INTERP_SHIFT_PS1087
- psrad xm5, INTERP_SHIFT_PS1088
- packssdw xm4, xm51089
-1090
- movu [r2], xm41091
+ PROCESS_IPFILTER_LUMA_PS_8x1_AVX2 81092
add r2, r31093
add r0, r11094
dec r4d1095
1096
mov r5d, r5m1097
shl r4d, 41098
%ifdef PIC1099
- lea r6, [tab_LumaCoeff]1100
+ lea r6, [h_tab_LumaCoeff]1101
vpbroadcastq m0, [r6 + r4]1102
vpbroadcastq m1, [r6 + r4 + 8]1103
%else1104
- vpbroadcastq m0, [tab_LumaCoeff + r4]1105
- vpbroadcastq m1, [tab_LumaCoeff + r4 + 8]1106
+ vpbroadcastq m0, [h_tab_LumaCoeff + r4]1107
+ vpbroadcastq m1, [h_tab_LumaCoeff + r4 + 8]1108
%endif1109
mova m3, [interp8_hpp_shuf]1110
vbroadcasti128 m2, [INTERP_OFFSET_PS]1111
1112
sub r0, r61113
add r4d, 71114
1115
+1116
.loop0:1117
-%assign x 01118
-%rep 24/81119
- vbroadcasti128 m4, [r0 + x]1120
- vbroadcasti128 m5, [r0 + 8 + x]1121
- pshufb m4, m31122
- pshufb m7, m5, m31123
- pmaddwd m4, m01124
- pmaddwd m7, m11125
- paddd m4, m71126
+ PROCESS_IPFILTER_LUMA_PS_8x1_AVX2 241127
+ add r2, r31128
+ add r0, r11129
+ dec r4d1130
+ jnz .loop01131
+ RET1132
+%endif1133
1134
- vbroadcasti128 m6, [r0 + 16 + x]1135
- pshufb m5, m31136
- pshufb m6, m31137
- pmaddwd m5, m01138
- pmaddwd m6, m11139
- paddd m5, m61140
1141
- phaddd m4, m51142
- vpermq m4, m4, q31201143
- paddd m4, m21144
- vextracti128 xm5,m4, 11145
- psrad xm4, INTERP_SHIFT_PS1146
- psrad xm5, INTERP_SHIFT_PS1147
- packssdw xm4, xm51148
+%macro PROCESS_IPFILTER_LUMA_PS_16x1_AVX2 01149
+ movu m7, [r0]1150
+ movu m8, [r0 + 8]1151
+ pshufb m10, m7, m141152
+ pshufb m7, m131153
+ pshufb m11, m8, m141154
+ pshufb m8, m131155
+1156
+ pmaddwd m7, m01157
+ pmaddwd m10, m11158
+ paddd m7, m101159
+ pmaddwd m10, m11, m31160
+ pmaddwd m9, m8, m21161
+ paddd m10, m91162
+ paddd m7, m101163
+ paddd m7, m41164
+ psrad m7, INTERP_SHIFT_PS1165
+ movu m9, [r0 + 16]1166
+ pshufb m10, m9, m141167
+ pshufb m9, m131168
+ pmaddwd m8, m01169
+ pmaddwd m11, m11170
+ paddd m8, m111171
+ pmaddwd m10, m31172
+ pmaddwd m9, m21173
+ paddd m9, m101174
+ paddd m8, m91175
+ paddd m8, m41176
+ psrad m8, INTERP_SHIFT_PS1177
+ packssdw m7, m81178
+ pshufb m7, m121179
+ movu [r2], m71180
+%endmacro1181
1182
- movu [r2 + x], xm41183
- %assign x x+161184
- %endrep1185
+%macro IPFILTER_LUMA_PS_16xN_AVX2 11186
+INIT_YMM avx21187
+%if ARCH_X86_64 == 11188
+cglobal interp_8tap_horiz_ps_16x%1, 5, 6, 151189
1190
- add r2, r31191
- add r0, r11192
+ shl r1d, 11193
+ shl r3d, 11194
+ mov r4d, r4m1195
+ mov r5d, r5m1196
+ shl r4d, 41197
+%ifdef PIC1198
+ lea r6, [h_tab_LumaCoeff]1199
+ vpbroadcastd m0, [r6 + r4]1200
+ vpbroadcastd m1, [r6 + r4 + 4]1201
+ vpbroadcastd m2, [r6 + r4 + 8]1202
+ vpbroadcastd m3, [r6 + r4 + 12]1203
+%else1204
+ vpbroadcastd m0, [h_tab_LumaCoeff + r4]1205
+ vpbroadcastd m1, [h_tab_LumaCoeff + r4 + 4]1206
+ vpbroadcastd m2, [h_tab_LumaCoeff + r4 + 8]1207
+ vpbroadcastd m3, [h_tab_LumaCoeff + r4 + 12]1208
+%endif1209
+ mova m13, [interp8_hpp_shuf1_load_avx512]1210
+ mova m14, [interp8_hpp_shuf2_load_avx512]1211
+ mova m12, [interp8_hpp_shuf1_store_avx512]1212
+ vbroadcasti128 m4, [INTERP_OFFSET_PS]1213
+1214
+ ; register map1215
+ ; m0 , m1 interpolate coeff1216
+1217
+ sub r0, 61218
+ test r5d, r5d1219
+ mov r4d, %11220
+ jz .loop01221
+ lea r6, [r1*3]1222
+ sub r0, r61223
+ add r4d, 71224
+1225
+.loop0:1226
+1227
+ PROCESS_IPFILTER_LUMA_PS_16x1_AVX21228
+ lea r0, [r0 + r1]1229
+ lea r2, [r2 + r3]1230
+ ;add r2, r31231
+ ;add r0, r11232
dec r4d1233
jnz .loop01234
RET1235
%endif1236
-%macro IPFILTER_LUMA_PS_32_64_AVX2 21237
+%endmacro1238
+1239
+ IPFILTER_LUMA_PS_16xN_AVX2 41240
+ IPFILTER_LUMA_PS_16xN_AVX2 81241
+ IPFILTER_LUMA_PS_16xN_AVX2 121242
+ IPFILTER_LUMA_PS_16xN_AVX2 161243
+ IPFILTER_LUMA_PS_16xN_AVX2 321244
+ IPFILTER_LUMA_PS_16xN_AVX2 641245
+%macro PROCESS_IPFILTER_LUMA_PS_32x1_AVX2 01246
+ PROCESS_IPFILTER_LUMA_PS_16x1_AVX21247
+ movu m7, [r0 + mmsize]1248
+ movu m8, [r0 + 8+ mmsize]1249
+ pshufb m10, m7, m141250
+ pshufb m7, m131251
+ pshufb m11, m8, m141252
+ pshufb m8, m131253
+1254
+ pmaddwd m7, m01255
+ pmaddwd m10, m11256
+ paddd m7, m101257
+ pmaddwd m10, m11, m31258
+ pmaddwd m9, m8, m21259
+ paddd m10, m91260
+ paddd m7, m101261
+ paddd m7, m41262
+ psrad m7, INTERP_SHIFT_PS1263
+ movu m9, [r0 + 16+ mmsize]1264
+ pshufb m10, m9, m141265
+ pshufb m9, m131266
+ pmaddwd m8, m01267
+ pmaddwd m11, m11268
+ paddd m8, m111269
+ pmaddwd m10, m31270
+ pmaddwd m9, m21271
+ paddd m9, m101272
+ paddd m8, m91273
+ paddd m8, m41274
+ psrad m8, INTERP_SHIFT_PS1275
+ packssdw m7, m81276
+ pshufb m7, m121277
+ movu [r2+ mmsize], m71278
+%endmacro1279
+1280
+%macro IPFILTER_LUMA_PS_32xN_AVX2 11281
+INIT_YMM avx21282
+%if ARCH_X86_641283
+cglobal interp_8tap_horiz_ps_32x%1, 5, 6, 151284
+1285
+ shl r1d, 11286
+ shl r3d, 11287
+ mov r4d, r4m1288
+ mov r5d, r5m1289
+ shl r4d, 41290
+%ifdef PIC1291
+ lea r6, [h_tab_LumaCoeff]1292
+ vpbroadcastd m0, [r6 + r4]1293
+ vpbroadcastd m1, [r6 + r4 + 4]1294
+ vpbroadcastd m2, [r6 + r4 + 8]1295
+ vpbroadcastd m3, [r6 + r4 + 12]1296
+%else1297
+ vpbroadcastd m0, [h_tab_LumaCoeff + r4]1298
+ vpbroadcastd m1, [h_tab_LumaCoeff + r4 + 4]1299
+ vpbroadcastd m2, [h_tab_LumaCoeff + r4 + 8]1300
+ vpbroadcastd m3, [h_tab_LumaCoeff + r4 + 12]1301
+%endif1302
+ mova m13, [interp8_hpp_shuf1_load_avx512]1303
+ mova m14, [interp8_hpp_shuf2_load_avx512]1304
+ mova m12, [interp8_hpp_shuf1_store_avx512]1305
+ vbroadcasti128 m4, [INTERP_OFFSET_PS]1306
+1307
+ ; register map1308
+ ; m0 , m1 interpolate coeff1309
+1310
+ sub r0, 61311
+ test r5d, r5d1312
+ mov r4d, %11313
+ jz .loop01314
+ lea r6, [r1*3]1315
+ sub r0, r61316
+ add r4d, 71317
+1318
+.loop0:1319
+ PROCESS_IPFILTER_LUMA_PS_32x1_AVX21320
+ lea r0, [r0 + r1]1321
+ lea r2, [r2 + r3]1322
+ ;add r2, r31323
+ ;add r0, r11324
+ dec r4d1325
+ jnz .loop01326
+ RET1327
+%endif1328
+%endmacro1329
+1330
+ IPFILTER_LUMA_PS_32xN_AVX2 81331
+ IPFILTER_LUMA_PS_32xN_AVX2 161332
+ IPFILTER_LUMA_PS_32xN_AVX2 241333
+ IPFILTER_LUMA_PS_32xN_AVX2 321334
+ IPFILTER_LUMA_PS_32xN_AVX2 641335
+1336
+%macro PROCESS_IPFILTER_LUMA_PS_64x1_AVX2 01337
+ PROCESS_IPFILTER_LUMA_PS_16x1_AVX21338
+%assign x 321339
+%rep 31340
+ movu m7, [r0 + x]1341
+ movu m8, [r0 + 8+ x]1342
+ pshufb m10, m7, m141343
+ pshufb m7, m131344
+ pshufb m11, m8, m141345
+ pshufb m8, m131346
+1347
+ pmaddwd m7, m01348
+ pmaddwd m10, m11349
+ paddd m7, m101350
+ pmaddwd m10, m11, m31351
+ pmaddwd m9, m8, m21352
+ paddd m10, m91353
+ paddd m7, m101354
+ paddd m7, m41355
+ psrad m7, INTERP_SHIFT_PS1356
+ movu m9, [r0 + 16+ x]1357
+ pshufb m10, m9, m141358
+ pshufb m9, m131359
+ pmaddwd m8, m01360
+ pmaddwd m11, m11361
+ paddd m8, m111362
+ pmaddwd m10, m31363
+ pmaddwd m9, m21364
+ paddd m9, m101365
+ paddd m8, m91366
+ paddd m8, m41367
+ psrad m8, INTERP_SHIFT_PS1368
+ packssdw m7, m81369
+ pshufb m7, m121370
+ movu [r2+ x], m71371
+%assign x x+321372
+%endrep1373
+%endmacro1374
+1375
+%macro IPFILTER_LUMA_PS_64xN_AVX2 11376
+INIT_YMM avx21377
+%if ARCH_X86_641378
+cglobal interp_8tap_horiz_ps_64x%1, 5, 6, 151379
+1380
+ shl r1d, 11381
+ shl r3d, 11382
+ mov r4d, r4m1383
+ mov r5d, r5m1384
+ shl r4d, 41385
+%ifdef PIC1386
+ lea r6, [h_tab_LumaCoeff]1387
+ vpbroadcastd m0, [r6 + r4]1388
+ vpbroadcastd m1, [r6 + r4 + 4]1389
+ vpbroadcastd m2, [r6 + r4 + 8]1390
+ vpbroadcastd m3, [r6 + r4 + 12]1391
+%else1392
+ vpbroadcastd m0, [h_tab_LumaCoeff + r4]1393
+ vpbroadcastd m1, [h_tab_LumaCoeff + r4 + 4]1394
+ vpbroadcastd m2, [h_tab_LumaCoeff + r4 + 8]1395
+ vpbroadcastd m3, [h_tab_LumaCoeff + r4 + 12]1396
+%endif1397
+ mova m13, [interp8_hpp_shuf1_load_avx512]1398
+ mova m14, [interp8_hpp_shuf2_load_avx512]1399
+ mova m12, [interp8_hpp_shuf1_store_avx512]1400
+ vbroadcasti128 m4, [INTERP_OFFSET_PS]1401
+1402
+ ; register map1403
+ ; m0 , m1 interpolate coeff1404
+1405
+ sub r0, 61406
+ test r5d, r5d1407
+ mov r4d, %11408
+ jz .loop01409
+ lea r6, [r1*3]1410
+ sub r0, r61411
+ add r4d, 71412
+1413
+.loop0:1414
+ PROCESS_IPFILTER_LUMA_PS_64x1_AVX21415
+ lea r0, [r0 + r1]1416
+ lea r2, [r2 + r3]1417
+ ;add r2, r31418
+ ;add r0, r11419
+ dec r4d1420
+ jnz .loop01421
+ RET1422
+%endif1423
+%endmacro1424
+1425
+ IPFILTER_LUMA_PS_64xN_AVX2 161426
+ IPFILTER_LUMA_PS_64xN_AVX2 321427
+ IPFILTER_LUMA_PS_64xN_AVX2 481428
+ IPFILTER_LUMA_PS_64xN_AVX2 641429
+1430
+%macro IPFILTER_LUMA_PS_48xN_AVX2 11431
INIT_YMM avx21432
%if ARCH_X86_64 == 11433
-cglobal interp_8tap_horiz_ps_%1x%2, 4, 6, 81434
+cglobal interp_8tap_horiz_ps_48x%1, 5, 9,151435
1436
add r1d, r1d1437
add r3d, r3d1438
1439
1440
sub r0, 61441
test r5d, r5d1442
- mov r4d, %21443
+ mov r4d, %11444
jz .loop01445
lea r6, [r1*3]1446
sub r0, r61447
1448
1449
.loop0:1450
%assign x 01451
-%rep %1/161452
+%rep 31453
vbroadcasti128 m4, [r0 + x]1454
vbroadcasti128 m5, [r0 + 4 * SIZEOF_PIXEL + x]1455
pshufb m4, m31456
1457
RET1458
%endif1459
%endmacro1460
-1461
- IPFILTER_LUMA_PS_32_64_AVX2 32, 81462
- IPFILTER_LUMA_PS_32_64_AVX2 32, 161463
- IPFILTER_LUMA_PS_32_64_AVX2 32, 241464
- IPFILTER_LUMA_PS_32_64_AVX2 32, 321465
- IPFILTER_LUMA_PS_32_64_AVX2 32, 641466
-1467
- IPFILTER_LUMA_PS_32_64_AVX2 64, 161468
- IPFILTER_LUMA_PS_32_64_AVX2 64, 321469
- IPFILTER_LUMA_PS_32_64_AVX2 64, 481470
- IPFILTER_LUMA_PS_32_64_AVX2 64, 641471
-1472
- IPFILTER_LUMA_PS_32_64_AVX2 48, 641473
-1474
-%macro IPFILTER_LUMA_PS_16xN_AVX2 11475
-INIT_YMM avx21476
-%if ARCH_X86_64 == 11477
-cglobal interp_8tap_horiz_ps_16x%1, 4, 6, 81478
-1479
- add r1d, r1d1480
- add r3d, r3d1481
- mov r4d, r4m1482
- mov r5d, r5m1483
- shl r4d, 41484
-%ifdef PIC1485
- lea r6, [tab_LumaCoeff]1486
- vpbroadcastq m0, [r6 + r4]1487
- vpbroadcastq m1, [r6 + r4 + 8]1488
-%else1489
- vpbroadcastq m0, [tab_LumaCoeff + r4]1490
- vpbroadcastq m1, [tab_LumaCoeff + r4 + 8]1491
-%endif1492
- mova m3, [interp8_hpp_shuf]1493
- vbroadcasti128 m2, [INTERP_OFFSET_PS]1494
-1495
- ; register map1496
- ; m0 , m1 interpolate coeff1497
-1498
- sub r0, 61499
- test r5d, r5d1500
- mov r4d, %11501
- jz .loop01502
- lea r6, [r1*3]1503
- sub r0, r61504
- add r4d, 71505
-1506
-.loop0:1507
- vbroadcasti128 m4, [r0]1508
- vbroadcasti128 m5, [r0 + 8]1509
- pshufb m4, m31510
- pshufb m7, m5, m31511
- pmaddwd m4, m01512
- pmaddwd m7, m11513
- paddd m4, m71514
-1515
- vbroadcasti128 m6, [r0 + 16]1516
- pshufb m5, m31517
- pshufb m7, m6, m31518
- pmaddwd m5, m01519
- pmaddwd m7, m11520
- paddd m5, m71521
-1522
- phaddd m4, m51523
- vpermq m4, m4, q31201524
- paddd m4, m21525
- vextracti128 xm5, m4, 11526
- psrad xm4, INTERP_SHIFT_PS1527
- psrad xm5, INTERP_SHIFT_PS1528
- packssdw xm4, xm51529
- movu [r2], xm41530
-1531
- vbroadcasti128 m5, [r0 + 24]1532
- pshufb m6, m31533
- pshufb m7, m5, m31534
- pmaddwd m6, m01535
- pmaddwd m7, m11536
- paddd m6, m71537
-1538
- vbroadcasti128 m7, [r0 + 32]1539
- pshufb m5, m31540
- pshufb m7, m31541
- pmaddwd m5, m01542
- pmaddwd m7, m11543
- paddd m5, m71544
-1545
- phaddd m6, m51546
- vpermq m6, m6, q31201547
- paddd m6, m21548
- vextracti128 xm5,m6, 11549
- psrad xm6, INTERP_SHIFT_PS1550
- psrad xm5, INTERP_SHIFT_PS1551
- packssdw xm6, xm51552
- movu [r2 + 16], xm61553
-1554
- add r2, r31555
- add r0, r11556
- dec r4d1557
- jnz .loop01558
- RET1559
-%endif1560
-%endmacro1561
-1562
- IPFILTER_LUMA_PS_16xN_AVX2 41563
- IPFILTER_LUMA_PS_16xN_AVX2 81564
- IPFILTER_LUMA_PS_16xN_AVX2 121565
- IPFILTER_LUMA_PS_16xN_AVX2 161566
- IPFILTER_LUMA_PS_16xN_AVX2 321567
- IPFILTER_LUMA_PS_16xN_AVX2 641568
-1569
+ IPFILTER_LUMA_PS_48xN_AVX2 641570
INIT_YMM avx21571
%if ARCH_X86_64 == 11572
cglobal interp_8tap_horiz_ps_12x16, 4, 6, 81573
1574
mov r5d, r5m1575
shl r4d, 41576
%ifdef PIC1577
- lea r6, [tab_LumaCoeff]1578
+ lea r6, [h_tab_LumaCoeff]1579
vpbroadcastq m0, [r6 + r4]1580
vpbroadcastq m1, [r6 + r4 + 8]1581
%else1582
- vpbroadcastq m0, [tab_LumaCoeff + r4]1583
- vpbroadcastq m1, [tab_LumaCoeff + r4 + 8]1584
+ vpbroadcastq m0, [h_tab_LumaCoeff + r4]1585
+ vpbroadcastq m1, [h_tab_LumaCoeff + r4 + 8]1586
%endif1587
mova m3, [interp8_hpp_shuf]1588
vbroadcasti128 m2, [INTERP_OFFSET_PS]1589
x265_2.7.tar.gz/source/common/x86/h4-ipfilter16.asm -> x265_2.9.tar.gz/source/common/x86/h4-ipfilter16.asm
Changed
299
1
2
3
tab_Tm16: db 0, 1, 2, 3, 4, 5, 6, 7, 2, 3, 4, 5, 6, 7, 8, 94
5
-tab_ChromaCoeff: dw 0, 64, 0, 06
+h4_tab_ChromaCoeff: dw 0, 64, 0, 07
dw -2, 58, 10, -28
dw -4, 54, 16, -29
dw -6, 46, 28, -410
11
add r4d, r4d12
13
%ifdef PIC14
- lea r6, [tab_ChromaCoeff]15
+ lea r6, [h4_tab_ChromaCoeff]16
movddup m0, [r6 + r4 * 4]17
%else18
- movddup m0, [tab_ChromaCoeff + r4 * 4]19
+ movddup m0, [h4_tab_ChromaCoeff + r4 * 4]20
%endif21
22
%ifidn %3, ps23
24
; void interp_4tap_horiz_pp_%1x%2(pixel *src, intptr_t srcStride, pixel *dst, intptr_t dstStride, int coeffIdx)25
;-----------------------------------------------------------------------------26
27
+%if ARCH_X86_6428
FILTER_HOR_CHROMA_sse3 2, 4, pp29
FILTER_HOR_CHROMA_sse3 2, 8, pp30
FILTER_HOR_CHROMA_sse3 2, 16, pp31
32
FILTER_HOR_CHROMA_sse3 64, 32, ps33
FILTER_HOR_CHROMA_sse3 64, 48, ps34
FILTER_HOR_CHROMA_sse3 64, 64, ps35
+%endif36
37
%macro FILTER_W2_2 138
movu m3, [r0]39
40
add r4d, r4d41
42
%ifdef PIC43
- lea r%6, [tab_ChromaCoeff]44
+ lea r%6, [h4_tab_ChromaCoeff]45
movh m0, [r%6 + r4 * 4]46
%else47
- movh m0, [tab_ChromaCoeff + r4 * 4]48
+ movh m0, [h4_tab_ChromaCoeff + r4 * 4]49
%endif50
51
punpcklqdq m0, m052
53
add r4d, r4d54
55
%ifdef PIC56
- lea r%4, [tab_ChromaCoeff]57
+ lea r%4, [h4_tab_ChromaCoeff]58
movh m0, [r%4 + r4 * 4]59
%else60
- movh m0, [tab_ChromaCoeff + r4 * 4]61
+ movh m0, [h4_tab_ChromaCoeff + r4 * 4]62
%endif63
64
punpcklqdq m0, m065
66
sub r0, 267
mov r4d, r4m68
%ifdef PIC69
- lea r5, [tab_ChromaCoeff]70
+ lea r5, [h4_tab_ChromaCoeff]71
vpbroadcastq m0, [r5 + r4 * 8]72
%else73
- vpbroadcastq m0, [tab_ChromaCoeff + r4 * 8]74
+ vpbroadcastq m0, [h4_tab_ChromaCoeff + r4 * 8]75
%endif76
mova m1, [h4_interp8_hpp_shuf]77
vpbroadcastd m2, [pd_32]78
79
sub r0, 280
mov r4d, r4m81
%ifdef PIC82
- lea r5, [tab_ChromaCoeff]83
+ lea r5, [h4_tab_ChromaCoeff]84
vpbroadcastq m0, [r5 + r4 * 8]85
%else86
- vpbroadcastq m0, [tab_ChromaCoeff + r4 * 8]87
+ vpbroadcastq m0, [h4_tab_ChromaCoeff + r4 * 8]88
%endif89
mova m1, [h4_interp8_hpp_shuf]90
vpbroadcastd m2, [pd_32]91
92
sub r0, 293
mov r4d, r4m94
%ifdef PIC95
- lea r5, [tab_ChromaCoeff]96
+ lea r5, [h4_tab_ChromaCoeff]97
vpbroadcastq m0, [r5 + r4 * 8]98
%else99
- vpbroadcastq m0, [tab_ChromaCoeff + r4 * 8]100
+ vpbroadcastq m0, [h4_tab_ChromaCoeff + r4 * 8]101
%endif102
mova m1, [h4_interp8_hpp_shuf]103
vpbroadcastd m2, [pd_32]104
105
sub r0, 2106
mov r4d, r4m107
%ifdef PIC108
- lea r5, [tab_ChromaCoeff]109
+ lea r5, [h4_tab_ChromaCoeff]110
vpbroadcastq m0, [r5 + r4 * 8]111
%else112
- vpbroadcastq m0, [tab_ChromaCoeff + r4 * 8]113
+ vpbroadcastq m0, [h4_tab_ChromaCoeff + r4 * 8]114
%endif115
mova m1, [h4_interp8_hpp_shuf]116
vpbroadcastd m2, [pd_32]117
118
sub r0, 2119
mov r4d, r4m120
%ifdef PIC121
- lea r5, [tab_ChromaCoeff]122
+ lea r5, [h4_tab_ChromaCoeff]123
vpbroadcastq m0, [r5 + r4 * 8]124
%else125
- vpbroadcastq m0, [tab_ChromaCoeff + r4 * 8]126
+ vpbroadcastq m0, [h4_tab_ChromaCoeff + r4 * 8]127
%endif128
mova m1, [h4_interp8_hpp_shuf]129
vpbroadcastd m2, [pd_32]130
131
sub r0, 2132
mov r4d, r4m133
%ifdef PIC134
- lea r5, [tab_ChromaCoeff]135
+ lea r5, [h4_tab_ChromaCoeff]136
vpbroadcastq m0, [r5 + r4 * 8]137
%else138
- vpbroadcastq m0, [tab_ChromaCoeff + r4 * 8]139
+ vpbroadcastq m0, [h4_tab_ChromaCoeff + r4 * 8]140
%endif141
mova m1, [h4_interp8_hpp_shuf]142
vpbroadcastd m2, [pd_32]143
144
sub r0, 2145
mov r4d, r4m146
%ifdef PIC147
- lea r5, [tab_ChromaCoeff]148
+ lea r5, [h4_tab_ChromaCoeff]149
vpbroadcastq m0, [r5 + r4 * 8]150
%else151
- vpbroadcastq m0, [tab_ChromaCoeff + r4 * 8]152
+ vpbroadcastq m0, [h4_tab_ChromaCoeff + r4 * 8]153
%endif154
mova m1, [h4_interp8_hpp_shuf]155
vpbroadcastd m2, [pd_32]156
157
sub r0, 2158
mov r4d, r4m159
%ifdef PIC160
- lea r5, [tab_ChromaCoeff]161
+ lea r5, [h4_tab_ChromaCoeff]162
vpbroadcastq m0, [r5 + r4 * 8]163
%else164
- vpbroadcastq m0, [tab_ChromaCoeff + r4 * 8]165
+ vpbroadcastq m0, [h4_tab_ChromaCoeff + r4 * 8]166
%endif167
mova m1, [h4_interp8_hpp_shuf]168
vpbroadcastd m2, [pd_32]169
170
sub r0, 2171
mov r4d, r4m172
%ifdef PIC173
- lea r5, [tab_ChromaCoeff]174
+ lea r5, [h4_tab_ChromaCoeff]175
vpbroadcastq m0, [r5 + r4 * 8]176
%else177
- vpbroadcastq m0, [tab_ChromaCoeff + r4 * 8]178
+ vpbroadcastq m0, [h4_tab_ChromaCoeff + r4 * 8]179
%endif180
mova m1, [h4_interp8_hpp_shuf]181
vpbroadcastd m2, [pd_32]182
183
sub r0, 2184
mov r4d, r4m185
%ifdef PIC186
- lea r5, [tab_ChromaCoeff]187
+ lea r5, [h4_tab_ChromaCoeff]188
vpbroadcastq m0, [r5 + r4 * 8]189
%else190
- vpbroadcastq m0, [tab_ChromaCoeff + r4 * 8]191
+ vpbroadcastq m0, [h4_tab_ChromaCoeff + r4 * 8]192
%endif193
mova m1, [h4_interp8_hpp_shuf]194
vpbroadcastd m2, [pd_32]195
196
mov r5d, r5m197
198
%ifdef PIC199
- lea r6, [tab_ChromaCoeff]200
+ lea r6, [h4_tab_ChromaCoeff]201
vpbroadcastq m0, [r6 + r4 * 8]202
%else203
- vpbroadcastq m0, [tab_ChromaCoeff + r4 * 8]204
+ vpbroadcastq m0, [h4_tab_ChromaCoeff + r4 * 8]205
%endif206
mova m3, [h4_interp8_hpp_shuf]207
vbroadcasti128 m2, [INTERP_OFFSET_PS]208
209
mov r5d, r5m210
211
%ifdef PIC212
- lea r6, [tab_ChromaCoeff]213
+ lea r6, [h4_tab_ChromaCoeff]214
vpbroadcastq m0, [r6 + r4 * 8]215
%else216
- vpbroadcastq m0, [tab_ChromaCoeff + r4 * 8]217
+ vpbroadcastq m0, [h4_tab_ChromaCoeff + r4 * 8]218
%endif219
mova m3, [h4_interp8_hpp_shuf]220
vbroadcasti128 m2, [INTERP_OFFSET_PS]221
222
mov r5d, r5m223
224
%ifdef PIC225
- lea r6, [tab_ChromaCoeff]226
+ lea r6, [h4_tab_ChromaCoeff]227
vpbroadcastq m0, [r6 + r4 * 8]228
%else229
- vpbroadcastq m0, [tab_ChromaCoeff + r4 * 8]230
+ vpbroadcastq m0, [h4_tab_ChromaCoeff + r4 * 8]231
%endif232
mova m3, [h4_interp8_hpp_shuf]233
vbroadcasti128 m2, [INTERP_OFFSET_PS]234
235
mov r5d, r5m236
237
%ifdef PIC238
- lea r6, [tab_ChromaCoeff]239
+ lea r6, [h4_tab_ChromaCoeff]240
vpbroadcastq m0, [r6 + r4 * 8]241
%else242
- vpbroadcastq m0, [tab_ChromaCoeff + r4 * 8]243
+ vpbroadcastq m0, [h4_tab_ChromaCoeff + r4 * 8]244
%endif245
mova m3, [h4_interp8_hpp_shuf]246
vbroadcasti128 m2, [INTERP_OFFSET_PS]247
248
mov r5d, r5m249
250
%ifdef PIC251
- lea r6, [tab_ChromaCoeff]252
+ lea r6, [h4_tab_ChromaCoeff]253
vpbroadcastq m0, [r6 + r4 * 8]254
%else255
- vpbroadcastq m0, [tab_ChromaCoeff + r4 * 8]256
+ vpbroadcastq m0, [h4_tab_ChromaCoeff + r4 * 8]257
%endif258
mova m3, [h4_interp8_hpp_shuf]259
vbroadcasti128 m2, [INTERP_OFFSET_PS]260
261
mov r5d, r5m262
263
%ifdef PIC264
- lea r6, [tab_ChromaCoeff]265
+ lea r6, [h4_tab_ChromaCoeff]266
vpbroadcastq m0, [r6 + r4 * 8]267
%else268
- vpbroadcastq m0, [tab_ChromaCoeff + r4 * 8]269
+ vpbroadcastq m0, [h4_tab_ChromaCoeff + r4 * 8]270
%endif271
mova m3, [h4_interp8_hpp_shuf]272
vbroadcasti128 m2, [INTERP_OFFSET_PS]273
274
mov r5d, r5m275
276
%ifdef PIC277
- lea r6, [tab_ChromaCoeff]278
+ lea r6, [h4_tab_ChromaCoeff]279
vpbroadcastq m0, [r6 + r4 * 8]280
%else281
- vpbroadcastq m0, [tab_ChromaCoeff + r4 * 8]282
+ vpbroadcastq m0, [h4_tab_ChromaCoeff + r4 * 8]283
%endif284
mova m3, [h4_interp8_hpp_shuf]285
vbroadcasti128 m2, [INTERP_OFFSET_PS]286
287
mov r5d, r5m288
289
%ifdef PIC290
- lea r6, [tab_ChromaCoeff]291
+ lea r6, [h4_tab_ChromaCoeff]292
vpbroadcastq m0, [r6 + r4 * 8]293
%else294
- vpbroadcastq m0, [tab_ChromaCoeff + r4 * 8]295
+ vpbroadcastq m0, [h4_tab_ChromaCoeff + r4 * 8]296
%endif297
mova m3, [h4_interp8_hpp_shuf]298
vbroadcasti128 m2, [INTERP_OFFSET_PS]299
x265_2.7.tar.gz/source/common/x86/intrapred.h -> x265_2.9.tar.gz/source/common/x86/intrapred.h
Changed
19
1
2
FUNCDEF_TU_S2(void, intra_pred_dc, sse2, pixel* dst, intptr_t dstStride, const pixel*srcPix, int, int filter);3
FUNCDEF_TU_S2(void, intra_pred_dc, sse4, pixel* dst, intptr_t dstStride, const pixel*srcPix, int, int filter);4
FUNCDEF_TU_S2(void, intra_pred_dc, avx2, pixel* dst, intptr_t dstStride, const pixel*srcPix, int, int filter);5
-6
+FUNCDEF_TU_S2(void, intra_pred_dc, avx512, pixel* dst, intptr_t dstStride, const pixel*srcPix, int, int filter);7
FUNCDEF_TU_S2(void, intra_pred_planar, sse2, pixel* dst, intptr_t dstStride, const pixel*srcPix, int, int filter);8
FUNCDEF_TU_S2(void, intra_pred_planar, sse4, pixel* dst, intptr_t dstStride, const pixel*srcPix, int, int filter);9
FUNCDEF_TU_S2(void, intra_pred_planar, avx2, pixel* dst, intptr_t dstStride, const pixel*srcPix, int, int filter);10
11
DECL_ALL(ssse3);12
DECL_ALL(sse4);13
DECL_ALL(avx2);14
-15
+DECL_ALL(avx512);16
#undef DECL_ALL17
#undef DECL_ANGS18
#undef DECL_ANG19
x265_2.7.tar.gz/source/common/x86/intrapred16.asm -> x265_2.9.tar.gz/source/common/x86/intrapred16.asm
Changed
2638
1
2
const pw_ang8_16, db 0, 0, 0, 0, 0, 0, 12, 13, 10, 11, 6, 7, 4, 5, 0, 13
const pw_ang8_17, db 0, 0, 14, 15, 12, 13, 10, 11, 8, 9, 4, 5, 2, 3, 0, 14
const pw_swap16, times 2 db 14, 15, 12, 13, 10, 11, 8, 9, 6, 7, 4, 5, 2, 3, 0, 15
-6
+const pw_swap16_avx512, times 4 db 14, 15, 12, 13, 10, 11, 8, 9, 6, 7, 4, 5, 2, 3, 0, 17
const pw_ang16_13, db 14, 15, 8, 9, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 08
const pw_ang16_16, db 0, 0, 0, 0, 0, 0, 10, 11, 8, 9, 6, 7, 2, 3, 0, 19
10
11
;-----------------------------------------------------------------------------------12
; void intra_pred_dc(pixel* dst, intptr_t dstStride, pixel* above, int, int filter)13
;-----------------------------------------------------------------------------------14
+%if ARCH_X86_6415
INIT_XMM sse216
cglobal intra_pred_dc8, 5, 8, 217
movu m0, [r2 + 34]18
19
mov [r0 + r7], r3w20
.end:21
RET22
+%endif23
24
;-------------------------------------------------------------------------------------------------------25
; void intra_pred_dc(pixel* dst, intptr_t dstStride, pixel* left, pixel* above, int dirMode, int filter)26
;-------------------------------------------------------------------------------------------------------27
+%if ARCH_X86_6428
+;This code is meant for 64 bit architecture29
INIT_XMM sse230
cglobal intra_pred_dc16, 5, 10, 431
lea r3, [r2 + 66]32
33
mov [r9 + r1 * 8], r3w34
.end:35
RET36
+%endif37
38
;-------------------------------------------------------------------------------------------39
; void intra_pred_dc(pixel* above, pixel* left, pixel* dst, intptr_t dstStride, int filter)40
41
;-------------------------------------------------------------------------------------------------------42
; void intra_pred_dc(pixel* dst, intptr_t dstStride, pixel* left, pixel* above, int dirMode, int filter)43
;-------------------------------------------------------------------------------------------------------44
+%if ARCH_X86_6445
INIT_YMM avx246
cglobal intra_pred_dc16, 3, 9, 447
mov r3d, r4m48
49
movu [r0 + r2 * 1 + 0], m050
movu [r0 + r2 * 1 + mmsize], m051
RET52
+INIT_ZMM avx51253
+cglobal intra_pred_dc32, 3,3,254
+ add r2, 255
+ add r1d, r1d56
+ movu m0, [r2]57
+ movu m1, [r2 + 2 * mmsize]58
+ paddw m0, m159
+ vextracti32x8 ym1, m0, 160
+ paddw ym0, ym161
+ vextracti32x4 xm1, m0, 162
+ paddw xm0, xm163
+ pmaddwd xm0, [pw_1]64
+ movhlps xm1, xm065
+ paddd xm0, xm166
+ vpsrldq xm1, xm0, 467
+ paddd xm0, xm168
+ paddd xm0, [pd_32] ; sum = sum + 3269
+ psrld xm0, 6 ; sum = sum / 6470
+ vpbroadcastw m0, xm071
+ lea r2, [r1 * 3]72
+ ; store DC 32x3273
+ movu [r0 + r1 * 0 + 0], m074
+ movu [r0 + r1 * 1 + 0], m075
+ movu [r0 + r1 * 2 + 0], m076
+ movu [r0 + r2 * 1 + 0], m077
+ lea r0, [r0 + r1 * 4]78
+ movu [r0 + r1 * 0 + 0], m079
+ movu [r0 + r1 * 1 + 0], m080
+ movu [r0 + r1 * 2 + 0], m081
+ movu [r0 + r2 * 1 + 0], m082
+ lea r0, [r0 + r1 * 4]83
+ movu [r0 + r1 * 0 + 0], m084
+ movu [r0 + r1 * 1 + 0], m085
+ movu [r0 + r1 * 2 + 0], m086
+ movu [r0 + r2 * 1 + 0], m087
+ lea r0, [r0 + r1 * 4]88
+ movu [r0 + r1 * 0 + 0], m089
+ movu [r0 + r1 * 1 + 0], m090
+ movu [r0 + r1 * 2 + 0], m091
+ movu [r0 + r2 * 1 + 0], m092
+ lea r0, [r0 + r1 * 4]93
+ movu [r0 + r1 * 0 + 0], m094
+ movu [r0 + r1 * 1 + 0], m095
+ movu [r0 + r1 * 2 + 0], m096
+ movu [r0 + r2 * 1 + 0], m097
+ lea r0, [r0 + r1 * 4]98
+ movu [r0 + r1 * 0 + 0], m099
+ movu [r0 + r1 * 1 + 0], m0100
+ movu [r0 + r1 * 2 + 0], m0101
+ movu [r0 + r2 * 1 + 0], m0102
+ lea r0, [r0 + r1 * 4]103
+ movu [r0 + r1 * 0 + 0], m0104
+ movu [r0 + r1 * 1 + 0], m0105
+ movu [r0 + r1 * 2 + 0], m0106
+ movu [r0 + r2 * 1 + 0], m0107
+ lea r0, [r0 + r1 * 4]108
+ movu [r0 + r1 * 0 + 0], m0109
+ movu [r0 + r1 * 1 + 0], m0110
+ movu [r0 + r1 * 2 + 0], m0111
+ movu [r0 + r2 * 1 + 0], m0112
+ RET113
+%endif114
115
;---------------------------------------------------------------------------------------116
; void intra_pred_planar(pixel* dst, intptr_t dstStride, pixel*srcPix, int, int filter)117
118
;---------------------------------------------------------------------------------------119
; void intra_pred_planar(pixel* dst, intptr_t dstStride, pixel*srcPix, int, int filter)120
;---------------------------------------------------------------------------------------121
+%if ARCH_X86_64122
INIT_XMM sse2123
cglobal intra_pred_planar32, 3,3,16124
movd m3, [r2 + 66] ; topRight = above[32]125
126
%endrep127
RET128
%endif129
-130
+%endif131
;---------------------------------------------------------------------------------------132
; void intra_pred_planar(pixel* dst, intptr_t dstStride, pixel*srcPix, int, int filter)133
;---------------------------------------------------------------------------------------134
135
STORE_4x4136
RET137
138
+%if ARCH_X86_64139
cglobal intra_pred_ang4_26, 3,3,3140
movh m0, [r2 + 2] ;[8 7 6 5 4 3 2 1]141
add r1d, r1d142
143
mov [r0 + r3], r2w144
.quit:145
RET146
+%endif147
148
cglobal intra_pred_ang4_27, 3,3,5149
movu m0, [r2 + 2] ;[8 7 6 5 4 3 2 1]150
151
152
%macro TRANSPOSE_STORE_AVX2 11153
jnz .skip%11154
- punpckhwd m%9, m%1, m%2155
- punpcklwd m%1, m%2156
- punpckhwd m%2, m%3, m%4157
- punpcklwd m%3, m%4158
-159
- punpckldq m%4, m%1, m%3160
- punpckhdq m%1, m%3161
- punpckldq m%3, m%9, m%2162
- punpckhdq m%9, m%2163
-164
- punpckhwd m%10, m%5, m%6165
- punpcklwd m%5, m%6166
- punpckhwd m%6, m%7, m%8167
- punpcklwd m%7, m%8168
-169
- punpckldq m%8, m%5, m%7170
- punpckhdq m%5, m%7171
- punpckldq m%7, m%10, m%6172
- punpckhdq m%10, m%6173
-174
- punpcklqdq m%6, m%4, m%8175
- punpckhqdq m%2, m%4, m%8176
- punpcklqdq m%4, m%1, m%5177
- punpckhqdq m%8, m%1, m%5178
-179
- punpcklqdq m%1, m%3, m%7180
- punpckhqdq m%5, m%3, m%7181
- punpcklqdq m%3, m%9, m%10182
- punpckhqdq m%7, m%9, m%10183
+ punpckhwd ym%9, ym%1, ym%2184
+ punpcklwd ym%1, ym%2185
+ punpckhwd ym%2, ym%3, ym%4186
+ punpcklwd ym%3, ym%4187
+188
+ punpckldq ym%4, ym%1, ym%3189
+ punpckhdq ym%1, ym%3190
+ punpckldq ym%3, ym%9, ym%2191
+ punpckhdq ym%9, ym%2192
+193
+ punpckhwd ym%10, ym%5, ym%6194
+ punpcklwd ym%5, ym%6195
+ punpckhwd ym%6, ym%7, ym%8196
+ punpcklwd ym%7, ym%8197
+198
+ punpckldq ym%8, ym%5, ym%7199
+ punpckhdq ym%5, ym%7200
+ punpckldq ym%7, ym%10, ym%6201
+ punpckhdq ym%10, ym%6202
+203
+ punpcklqdq ym%6, ym%4, ym%8204
+ punpckhqdq ym%2, ym%4, ym%8205
+ punpcklqdq ym%4, ym%1, ym%5206
+ punpckhqdq ym%8, ym%1, ym%5207
+208
+ punpcklqdq ym%1, ym%3, ym%7209
+ punpckhqdq ym%5, ym%3, ym%7210
+ punpcklqdq ym%3, ym%9, ym%10211
+ punpckhqdq ym%7, ym%9, ym%10212
213
movu [r0 + r1 * 0 + %11], xm%6214
movu [r0 + r1 * 1 + %11], xm%2215
216
movu [r5 + r4 * 1 + %11], xm%7217
218
lea r5, [r5 + r1 * 4]219
- vextracti128 [r5 + r1 * 0 + %11], m%6, 1220
- vextracti128 [r5 + r1 * 1 + %11], m%2, 1221
- vextracti128 [r5 + r1 * 2 + %11], m%4, 1222
- vextracti128 [r5 + r4 * 1 + %11], m%8, 1223
+ vextracti128 [r5 + r1 * 0 + %11], ym%6, 1224
+ vextracti128 [r5 + r1 * 1 + %11], ym%2, 1225
+ vextracti128 [r5 + r1 * 2 + %11], ym%4, 1226
+ vextracti128 [r5 + r4 * 1 + %11], ym%8, 1227
228
lea r5, [r5 + r1 * 4]229
- vextracti128 [r5 + r1 * 0 + %11], m%1, 1230
- vextracti128 [r5 + r1 * 1 + %11], m%5, 1231
- vextracti128 [r5 + r1 * 2 + %11], m%3, 1232
- vextracti128 [r5 + r4 * 1 + %11], m%7, 1233
+ vextracti128 [r5 + r1 * 0 + %11], ym%1, 1234
+ vextracti128 [r5 + r1 * 1 + %11], ym%5, 1235
+ vextracti128 [r5 + r1 * 2 + %11], ym%3, 1236
+ vextracti128 [r5 + r4 * 1 + %11], ym%7, 1237
jmp .end%11238
.skip%11:239
- movu [r0 + r1 * 0], m%1240
- movu [r0 + r1 * 1], m%2241
- movu [r0 + r1 * 2], m%3242
- movu [r0 + r4 * 1], m%4243
+ movu [r0 + r1 * 0], ym%1244
+ movu [r0 + r1 * 1], ym%2245
+ movu [r0 + r1 * 2], ym%3246
+ movu [r0 + r4 * 1], ym%4247
248
lea r0, [r0 + r1 * 4]249
- movu [r0 + r1 * 0], m%5250
- movu [r0 + r1 * 1], m%6251
- movu [r0 + r1 * 2], m%7252
- movu [r0 + r4 * 1], m%8253
+ movu [r0 + r1 * 0], ym%5254
+ movu [r0 + r1 * 1], ym%6255
+ movu [r0 + r1 * 2], ym%7256
+ movu [r0 + r4 * 1], ym%8257
lea r0, [r0 + r1 * 4]258
.end%11:259
%endmacro260
261
+%if ARCH_X86_64262
;; angle 16, modes 3 and 33263
cglobal ang16_mode_3_33264
test r6d, r6d265
266
packusdw m11, m3267
TRANSPOSE_STORE_AVX2 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 16268
ret269
-270
;; angle 16, modes 7 and 29271
cglobal ang16_mode_7_29272
test r6d, r6d273
274
275
mov rsp, [rsp+4*mmsize]276
RET277
+%endif278
;-------------------------------------------------------------------------------------------------------279
; end of avx2 code for intra_pred_ang32 mode 2 to 34280
;-------------------------------------------------------------------------------------------------------281
+;-------------------------------------------------------------------------------------------------------282
+; avx512 code for intra_pred_ang32 mode 2 to 34 start283
+;-------------------------------------------------------------------------------------------------------284
+INIT_ZMM avx512285
+cglobal intra_pred_ang32_2, 3,5,3286
+ lea r4, [r2]287
+ add r2, 128288
+ cmp r3m, byte 34289
+ cmove r2, r4290
+ add r1d, r1d291
+ lea r3, [r1 * 3]292
+ movu m0, [r2 + 4]293
+ movu m1, [r2 + 20]294
+295
+ movu [r0], m0296
+ palignr m2, m1, m0, 2297
+ movu [r0 + r1], m2298
+ palignr m2, m1, m0, 4299
+ movu [r0 + r1 * 2], m2300
+ palignr m2, m1, m0, 6301
+ movu [r0 + r3], m2302
+303
+ lea r0, [r0 + r1 * 4]304
+ palignr m2, m1, m0, 8305
+ movu [r0], m2306
+ palignr m2, m1, m0, 10307
+ movu [r0 + r1], m2308
+ palignr m2, m1, m0, 12309
+ movu [r0 + r1 * 2], m2310
+ palignr m2, m1, m0, 14311
+ movu [r0 + r3], m2312
+313
+ movu m0, [r2 + 36]314
+ lea r0, [r0 + r1 * 4]315
+ movu [r0], m1316
+ palignr m2, m0, m1, 2317
+ movu [r0 + r1], m2318
+ palignr m2, m0, m1, 4319
+ movu [r0 + r1 * 2], m2320
+ palignr m2, m0, m1, 6321
+ movu [r0 + r3], m2322
+323
+ lea r0, [r0 + r1 * 4]324
+ palignr m2, m0, m1, 8325
+ movu [r0], m2326
+ palignr m2, m0, m1, 10327
+ movu [r0 + r1], m2328
+ palignr m2, m0, m1, 12329
+ movu [r0 + r1 * 2], m2330
+ palignr m2, m0, m1, 14331
+ movu [r0 + r3], m2332
+333
+ lea r0, [r0 + r1 * 4]334
+ movu m1, [r2 + 52]335
+336
+ movu [r0], m0337
+ palignr m2, m1, m0, 2338
+ movu [r0 + r1], m2339
+ palignr m2, m1, m0, 4340
+ movu [r0 + r1 * 2], m2341
+ palignr m2, m1, m0, 6342
+ movu [r0 + r3], m2343
+344
+ lea r0, [r0 + r1 * 4]345
+ palignr m2, m1, m0, 8346
+ movu [r0], m2347
+ palignr m2, m1, m0, 10348
+ movu [r0 + r1], m2349
+ palignr m2, m1, m0, 12350
+ movu [r0 + r1 * 2], m2351
+ palignr m2, m1, m0, 14352
+ movu [r0 + r3], m2353
+354
+ movu m0, [r2 + 68]355
+ lea r0, [r0 + r1 * 4]356
+ movu [r0], m1357
+ palignr m2, m0, m1, 2358
+ movu [r0 + r1], m2359
+ palignr m2, m0, m1, 4360
+ movu [r0 + r1 * 2], m2361
+ palignr m2, m0, m1, 6362
+ movu [r0 + r3], m2363
+364
+ lea r0, [r0 + r1 * 4]365
+ palignr m2, m0, m1, 8366
+ movu [r0], m2367
+ palignr m2, m0, m1, 10368
+ movu [r0 + r1], m2369
+ palignr m2, m0, m1, 12370
+ movu [r0 + r1 * 2], m2371
+ palignr m2, m0, m1, 14372
+ movu [r0 + r3], m2373
+ RET374
+375
+cglobal intra_pred_ang32_10, 3,4,2376
+ add r2, mmsize*2377
+ add r1d, r1d378
+ lea r3, [r1 * 3]379
+380
+ vpbroadcastw m0, [r2 + 2] ; [1...]381
+ vpbroadcastw m1, [r2 + 2 + 2] ; [2...]382
+ movu [r0], m0383
+ movu [r0 + r1], m1384
+385
+ vpbroadcastw m0, [r2 + 2 + 4] ; [3...]386
+ vpbroadcastw m1, [r2 + 2 + 6] ; [4...]387
+ movu [r0 + r1 * 2], m0388
+ movu [r0 + r3], m1389
+ lea r0, [r0 + r1 * 4]390
+391
+ vpbroadcastw m0, [r2 + 2 + 8] ; [5...]392
+ vpbroadcastw m1, [r2 + 2 + 10] ; [6...]393
+ movu [r0], m0394
+ movu [r0 + r1], m1395
+396
+ vpbroadcastw m0, [r2 + 2 + 12] ; [7...]397
+ vpbroadcastw m1, [r2 + 2 + 14] ; [8...]398
+ movu [r0 + r1 * 2], m0399
+ movu [r0 + r3], m1400
+ lea r0, [r0 + r1 *4]401
+402
+ vpbroadcastw m0, [r2 + 2 + 16] ; [9...]403
+ vpbroadcastw m1, [r2 + 2 + 18] ; [10...]404
+ movu [r0], m0405
+ movu [r0 + r1], m1406
+407
+ vpbroadcastw m0, [r2 + 2 + 20] ; [11...]408
+ vpbroadcastw m1, [r2 + 2 + 22] ; [12...]409
+ movu [r0 + r1 * 2], m0410
+ movu [r0 + r3], m1411
+ lea r0, [r0 + r1 *4]412
+413
+ vpbroadcastw m0, [r2 + 2 + 24] ; [13...]414
+ vpbroadcastw m1, [r2 + 2 + 26] ; [14...]415
+ movu [r0], m0416
+ movu [r0 + r1], m1417
+418
+ vpbroadcastw m0, [r2 + 2 + 28] ; [15...]419
+ vpbroadcastw m1, [r2 + 2 + 30] ; [16...]420
+ movu [r0 + r1 * 2], m0421
+ movu [r0 + r3], m1422
+ lea r0, [r0 + r1 *4]423
+424
+ vpbroadcastw m0, [r2 + 2 + 32] ; [17...]425
+ vpbroadcastw m1, [r2 + 2 + 34] ; [18...]426
+ movu [r0], m0427
+ movu [r0 + r1], m1428
+429
+ vpbroadcastw m0, [r2 + 2 + 36] ; [19...]430
+ vpbroadcastw m1, [r2 + 2 + 38] ; [20...]431
+ movu [r0 + r1 * 2], m0432
+ movu [r0 + r3], m1433
+ lea r0, [r0 + r1 *4]434
+435
+ vpbroadcastw m0, [r2 + 2 + 40] ; [21...]436
+ vpbroadcastw m1, [r2 + 2 + 42] ; [22...]437
+ movu [r0], m0438
+ movu [r0 + r1], m1439
+440
+ vpbroadcastw m0, [r2 + 2 + 44] ; [23...]441
+ vpbroadcastw m1, [r2 + 2 + 46] ; [24...]442
+ movu [r0 + r1 * 2], m0443
+ movu [r0 + r3], m1444
+ lea r0, [r0 + r1 *4]445
+446
+ vpbroadcastw m0, [r2 + 2 + 48] ; [25...]447
+ vpbroadcastw m1, [r2 + 2 + 50] ; [26...]448
+ movu [r0], m0449
+ movu [r0 + r1], m1450
+451
+ vpbroadcastw m0, [r2 + 2 + 52] ; [27...]452
+ vpbroadcastw m1, [r2 + 2 + 54] ; [28...]453
+ movu [r0 + r1 * 2], m0454
+ movu [r0 + r3], m1455
+ lea r0, [r0 + r1 *4]456
+457
+ vpbroadcastw m0, [r2 + 2 + 56] ; [29...]458
+ vpbroadcastw m1, [r2 + 2 + 58] ; [30...]459
+ movu [r0], m0460
+ movu [r0 + r1], m1461
+462
+ vpbroadcastw m0, [r2 + 2 + 60] ; [31...]463
+ vpbroadcastw m1, [r2 + 2 + 62] ; [32...]464
+ movu [r0 + r1 * 2], m0465
+ movu [r0 + r3], m1466
+ RET467
+468
+cglobal intra_pred_ang32_18, 3,6,6469
+ mov r4, rsp470
+ sub rsp, 4*(mmsize/2)+gprsize471
+ and rsp, ~63472
+ mov [rsp+4*(mmsize/2)], r4473
+474
+ movu m0, [r2]475
+ mova [rsp + 2*(mmsize/2)], ym0476
+ vextracti32x8 [rsp + 3*(mmsize/2)], m0, 1477
+478
+ movu m2, [r2 + 130]479
+ pshufb m2, [pw_swap16_avx512]480
+ vpermq m2, m2, q1032481
+ mova [rsp + 1*(mmsize/2)], ym2482
+ vextracti32x8 [rsp + 0*(mmsize/2)], m2, 1483
+484
+ add r1d, r1d485
+ lea r2, [rsp+2*(mmsize/2)]486
+ lea r4, [r1 * 2]487
+ lea r3, [r1 * 3]488
+ lea r5, [r1 * 4]489
+490
+ movu m0, [r2]491
+ movu m2, [r2 - 16]492
+ movu [r0], m0493
+494
+ palignr m4, m0, m2, 14495
+ palignr m5, m0, m2, 12496
+ movu [r0 + r1], m4497
+ movu [r0 + r4], m5498
+499
+ palignr m4, m0, m2, 10500
+ palignr m5, m0, m2, 8501
+ movu [r0 + r3], m4502
+ add r0, r5503
+ movu [r0], m5504
+505
+ palignr m4, m0, m2, 6506
+ palignr m5, m0, m2, 4507
+ movu [r0 + r1], m4508
+ movu [r0 + r4], m5509
+510
+ palignr m4, m0, m2, 2511
+ movu [r0 + r3], m4512
+ add r0, r5513
+ movu [r0], m2514
+515
+ movu m0, [r2 - 32]516
+ palignr m4, m2, m0, 14517
+ palignr m5, m2, m0, 12518
+ movu [r0 + r1], m4519
+ movu [r0 + r4], m5520
+521
+ palignr m4, m2, m0, 10522
+ palignr m5, m2, m0, 8523
+ movu [r0 + r3], m4524
+ add r0, r5525
+ movu [r0], m5526
+527
+ palignr m4, m2, m0, 6528
+ palignr m5, m2, m0, 4529
+ movu [r0 + r1], m4530
+ movu [r0 + r4], m5531
+532
+ palignr m4, m2, m0, 2533
+ movu [r0 + r3], m4534
+ add r0, r5535
+ movu [r0], m0536
+537
+ movu m2, [r2 - 48]538
+ palignr m4, m0, m2, 14539
+ palignr m5, m0, m2, 12540
+ movu [r0 + r1], m4541
+ movu [r0 + r4], m5542
+543
+ palignr m4, m0, m2, 10544
+ palignr m5, m0, m2, 8545
+ movu [r0 + r3], m4546
+ add r0, r5547
+ movu [r0], m5548
+549
+ palignr m4, m0, m2, 6550
+ palignr m5, m0, m2, 4551
+ movu [r0 + r1], m4552
+ movu [r0 + r4], m5553
+554
+ palignr m4, m0, m2, 2555
+ movu [r0 + r3], m4556
+ add r0, r5557
+ movu [r0], m2558
+559
+ movu m0, [r2 - 64]560
+ palignr m4, m2, m0, 14561
+ palignr m5, m2, m0, 12562
+ movu [r0 + r1], m4563
+ movu [r0 + r4], m5564
+565
+ palignr m4, m2, m0, 10566
+ palignr m5, m2, m0, 8567
+ movu [r0 + r3], m4568
+ add r0, r5569
+ movu [r0], m5570
+571
+ palignr m4, m2, m0, 6572
+ palignr m5, m2, m0, 4573
+ movu [r0 + r1], m4574
+ movu [r0 + r4], m5575
+576
+ palignr m4, m2, m0, 2577
+ movu [r0 + r3], m4578
+ mov rsp, [rsp+4*(mmsize/2)]579
+ RET580
+INIT_ZMM avx512581
+cglobal intra_pred_ang32_26, 3,3,2582
+ movu m0, [r2 + 2]583
+ add r1d, r1d584
+ lea r2, [r1 * 3]585
+ movu [r0], m0586
+ movu [r0 + r1], m0587
+ movu [r0 + r1 * 2], m0588
+ movu [r0 + r2], m0589
+ lea r0, [r0 + r1 *4]590
+ movu [r0], m0591
+ movu [r0 + r1], m0592
+ movu [r0 + r1 * 2], m0593
+ movu [r0 + r2], m0594
+ lea r0, [r0 + r1 *4]595
+ movu [r0], m0596
+ movu [r0 + r1], m0597
+ movu [r0 + r1 * 2], m0598
+ movu [r0 + r2], m0599
+ lea r0, [r0 + r1 *4]600
+ movu [r0], m0601
+ movu [r0 + r1], m0602
+ movu [r0 + r1 * 2], m0603
+ movu [r0 + r2], m0604
+ lea r0, [r0 + r1 *4]605
+ movu [r0], m0606
+ movu [r0 + r1], m0607
+ movu [r0 + r1 * 2], m0608
+ movu [r0 + r2], m0609
+ lea r0, [r0 + r1 *4]610
+ movu [r0], m0611
+ movu [r0 + r1], m0612
+ movu [r0 + r1 * 2], m0613
+ movu [r0 + r2], m0614
+ lea r0, [r0 + r1 *4]615
+ movu [r0], m0616
+ movu [r0 + r1], m0617
+ movu [r0 + r1 * 2], m0618
+ movu [r0 + r2], m0619
+ lea r0, [r0 + r1 *4]620
+ movu [r0], m0621
+ movu [r0 + r1], m0622
+ movu [r0 + r1 * 2], m0623
+ movu [r0 + r2], m0624
+ RET625
+626
+;; angle 16, modes 9 and 27627
+cglobal ang16_mode_9_27628
+ test r6d, r6d629
+630
+ vbroadcasti32x8 m0, [r2 + 2] ; [16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1]631
+ vbroadcasti32x8 m1, [r2 + 4] ; [17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2]632
+633
+ punpcklwd m3, m0, m1 ; [13 12 12 11 11 10 10 9 5 4 4 3 3 2 2 1]634
+ punpckhwd m0, m1 ; [17 16 16 15 15 14 14 13 9 8 8 7 7 6 6 5]635
+636
+ vbroadcasti32x8 m2, [r2 + 18] ; [24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9]637
+ vbroadcasti32x8 m4, [r2 + 20] ; [25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10]638
+ punpcklwd m2, m4 ; [21 20 20 19 19 18 18 17 13 12 12 11 11 10 10 9]639
+640
+ movu ym16, [r3 - 14 * 32] ; [2]641
+ vinserti32x8 m16, [r3 - 12 * 32], 1 ; [4]642
+ pmaddwd m4, m3, m16643
+ paddd m4, m15644
+ psrld m4, 5645
+ pmaddwd m5, m0, m16646
+ paddd m5, m15647
+ psrld m5, 5648
+ packusdw m4, m5649
+ vextracti32x8 ym5, m4, 1650
+ movu ym16, [r3 - 10 * 32] ; [6]651
+ vinserti32x8 m16, [r3 - 8 * 32], 1 ; [8]652
+ pmaddwd m6, m3, m16653
+ paddd m6, m15654
+ psrld m6, 5655
+ pmaddwd m9, m0, m16656
+ paddd m9, m15657
+ psrld m9, 5658
+ packusdw m6, m9659
+ vextracti32x8 ym7, m6, 1660
+ movu ym16, [r3 - 6 * 32] ; [10]661
+ vinserti32x8 m16, [r3 - 4 * 32], 1 ; [12]662
+ pmaddwd m8, m3, m16663
+ paddd m8, m15664
+ psrld m8, 5665
+ pmaddwd m9, m0, m16666
+ paddd m9, m15667
+ psrld m9, 5668
+ packusdw m8, m9669
+ vextracti32x8 ym9, m8, 1670
+ movu ym16, [r3 - 2 * 32] ; [14]671
+ vinserti32x8 m16, [r3], 1 ; [16]672
+ pmaddwd m10, m3, m16673
+ paddd m10, m15674
+ psrld m10, 5675
+ pmaddwd m1, m0, m16676
+ paddd m1, m15677
+ psrld m1, 5678
+ packusdw m10, m1679
+ vextracti32x8 ym11, m10, 1680
+ TRANSPOSE_STORE_AVX2 4, 5, 6, 7, 8, 9, 10, 11, 2, 1, 0681
+682
+ movu ym16, [r3 + 2 * 32] ; [18]683
+ vinserti32x8 m16, [r3 + 4 * 32], 1 ; [20]684
+ pmaddwd m4, m3, m16685
+ paddd m4, m15686
+ psrld m4, 5687
+ pmaddwd m5, m0, m16688
+ paddd m5, m15689
+ psrld m5, 5690
+ packusdw m4, m5691
+ vextracti32x8 ym5, m4, 1692
+ movu ym16, [r3 + 6 * 32] ; [22]693
+ vinserti32x8 m16, [r3 + 8 * 32], 1 ; [24]694
+ pmaddwd m6, m3, m16695
+ paddd m6, m15696
+ psrld m6, 5697
+ pmaddwd m8, m0, m16698
+ paddd m8, m15699
+ psrld m8, 5700
+ packusdw m6, m8701
+ vextracti32x8 ym7, m6, 1702
+ movu ym16, [r3 + 10 * 32] ; [26]703
+ vinserti32x8 m16, [r3 + 12 * 32], 1 ; [28]704
+ pmaddwd m8, m3, m16705
+ paddd m8, m15706
+ psrld m8, 5707
+ pmaddwd m9, m0, m16708
+ paddd m9, m15709
+ psrld m9, 5710
+ packusdw m8, m9711
+ vextracti32x8 ym9, m8, 1712
+ movu ym16, [r3 + 14 * 32] ; [30]713
+ pmaddwd ym3, ym16714
+ paddd ym3, ym15715
+ psrld ym3, 5716
+ pmaddwd ym0, ym16717
+ paddd ym0, ym15718
+ psrld ym0, 5719
+ packusdw ym3, ym0720
+721
+ movu ym1, [r2 + 4]722
+ TRANSPOSE_STORE_AVX2 4, 5, 6, 7, 8, 9, 3, 1, 0, 2, 16723
+ ret724
+725
+cglobal intra_pred_ang32_9, 3,8,17726
+ add r2, 128727
+ xor r6d, r6d728
+ lea r3, [ang_table_avx2 + 16 * 32]729
+ shl r1d, 1730
+ lea r4, [r1 * 3]731
+ lea r7, [r0 + 8 * r1]732
+ vbroadcasti32x8 m15, [pd_16]733
+734
+ call ang16_mode_9_27735
+ add r2, 2736
+ lea r0, [r0 + 32]737
+ call ang16_mode_9_27738
+ add r2, 30739
+ lea r0, [r7 + 8 * r1]740
+ call ang16_mode_9_27741
+ add r2, 2742
+ lea r0, [r0 + 32]743
+ call ang16_mode_9_27744
+ RET745
+746
+cglobal intra_pred_ang32_27, 3,7,17747
+ xor r6d, r6d748
+ inc r6d749
+ lea r3, [ang_table_avx2 + 16 * 32]750
+ shl r1d, 1751
+ lea r4, [r1 * 3]752
+ lea r5, [r0 + 32]753
+ vbroadcasti32x8 m15, [pd_16]754
+755
+ call ang16_mode_9_27756
+ add r2, 2757
+ call ang16_mode_9_27758
+ add r2, 30759
+ mov r0, r5760
+ call ang16_mode_9_27761
+ add r2, 2762
+ call ang16_mode_9_27763
+ RET764
+;; angle 16, modes 11 and 25765
+cglobal ang16_mode_11_25766
+ test r6d, r6d767
+768
+ vbroadcasti32x8 m0, [r2] ; [15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0]769
+ vbroadcasti32x8 m1, [r2 + 2] ; [16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1]770
+771
+ punpcklwd m3, m0, m1 ; [12 11 11 10 10 9 9 8 4 3 3 2 2 1 1 0]772
+ punpckhwd m0, m1 ; [16 15 15 14 14 13 13 12 8 7 7 6 6 5 5 4]773
+774
+ movu ym16, [r3 + 14 * 32] ; [30]775
+ vinserti32x8 m16, [r3 + 12 * 32], 1 ; [28]776
+ pmaddwd m4, m3, m16777
+ paddd m4, m15778
+ psrld m4, 5779
+ pmaddwd m5, m0, m16780
+ paddd m5, m15781
+ psrld m5, 5782
+ packusdw m4, m5783
+ vextracti32x8 ym5, m4, 1784
+ movu ym16, [r3 + 10 * 32] ; [26]785
+ vinserti32x8 m16, [r3 + 8 * 32], 1 ; [24]786
+ pmaddwd m6, m3, m16787
+ paddd m6, m15788
+ psrld m6, 5789
+ pmaddwd m9, m0, m16790
+ paddd m9, m15791
+ psrld m9, 5792
+ packusdw m6, m9793
+ vextracti32x8 ym7, m6, 1794
+ movu ym16, [r3 + 6 * 32] ; [22]795
+ vinserti32x8 m16, [r3 + 4 * 32], 1 ; [20]796
+ pmaddwd m8, m3, m16797
+ paddd m8, m15798
+ psrld m8, 5799
+ pmaddwd m9, m0, m16800
+ paddd m9, m15801
+ psrld m9, 5802
+ packusdw m8, m9803
+ vextracti32x8 ym9, m8, 1804
+ movu ym16, [r3 + 2 * 32] ; [18]805
+ vinserti32x8 m16, [r3], 1 ; [16]806
+ pmaddwd m10, m3, m16807
+ paddd m10, m15808
+ psrld m10, 5809
+ pmaddwd m1, m0, m16810
+ paddd m1, m15811
+ psrld m1, 5812
+ packusdw m10, m1813
+ vextracti32x8 ym11, m10, 1814
+ TRANSPOSE_STORE_AVX2 4, 5, 6, 7, 8, 9, 10, 11, 2, 1, 0815
+816
+ movu ym16, [r3 - 2 * 32] ; [14]817
+ vinserti32x8 m16, [r3 - 4 * 32], 1 ; [12]818
+ pmaddwd m4, m3, m16819
+ paddd m4, m15820
+ psrld m4, 5821
+ pmaddwd m5, m0, m16822
+ paddd m5, m15823
+ psrld m5, 5824
+ packusdw m4, m5825
+ vextracti32x8 ym5, m4, 1826
+ movu ym16, [r3 - 6 * 32] ; [10]827
+ vinserti32x8 m16, [r3 - 8 * 32], 1 ; [8]828
+ pmaddwd m6, m3, m16829
+ paddd m6, m15830
+ psrld m6, 5831
+ pmaddwd m8, m0, m16832
+ paddd m8, m15833
+ psrld m8, 5834
+ packusdw m6, m8835
+ vextracti32x8 ym7, m6, 1836
+ movu ym16, [r3 - 10 * 32] ; [6]837
+ vinserti32x8 m16, [r3 - 12 * 32], 1 ; [4]838
+ pmaddwd m8, m3, m16839
+ paddd m8, m15840
+ psrld m8, 5841
+ pmaddwd m9, m0, m16842
+ paddd m9, m15843
+ psrld m9, 5844
+ packusdw m8, m9845
+ vextracti32x8 ym9, m8, 1846
+ pmaddwd ym3, [r3 - 14 * 32] ; [2]847
+ paddd ym3, ym15848
+ psrld ym3, 5849
+ pmaddwd ym0, [r3 - 14 * 32]850
+ paddd ym0, ym15851
+ psrld ym0, 5852
+ packusdw ym3, ym0853
+854
+ movu ym1, [r2]855
+ TRANSPOSE_STORE_AVX2 4, 5, 6, 7, 8, 9, 3, 1, 0, 2, 16856
+ ret857
+858
+cglobal intra_pred_ang32_11, 3,8,17, 0-8859
+ movzx r5d, word [r2 + 128] ; [0]860
+ movzx r6d, word [r2]861
+ mov [rsp], r5w862
+ mov [r2 + 128], r6w863
+864
+ movzx r5d, word [r2 + 126] ; [16]865
+ movzx r6d, word [r2 + 32]866
+ mov [rsp + 4], r5w867
+ mov [r2 + 126], r6w868
+ vbroadcasti32x8 m15, [pd_16]869
+ add r2, 128870
+ xor r6d, r6d871
+ lea r3, [ang_table_avx2 + 16 * 32]872
+ shl r1d, 1873
+ lea r4, [r1 * 3]874
+ lea r7, [r0 + 8 * r1]875
+876
+ call ang16_mode_11_25877
+ sub r2, 2878
+ lea r0, [r0 + 32]879
+ call ang16_mode_11_25880
+ add r2, 34881
+ lea r0, [r7 + 8 * r1]882
+ call ang16_mode_11_25883
+ sub r2, 2884
+ lea r0, [r0 + 32]885
+ call ang16_mode_11_25886
+ mov r6d, [rsp]887
+ mov [r2 - 30], r6w888
+ mov r6d, [rsp + 4]889
+ mov [r2 - 32], r6w890
+ RET891
+892
+cglobal intra_pred_ang32_25, 3,7,17, 0-4893
+ xor r6d, r6d894
+ inc r6d895
+ lea r3, [ang_table_avx2 + 16 * 32]896
+ shl r1d, 1897
+ vbroadcasti32x8 m15, [pd_16]898
+ movzx r4d, word [r2 - 2]899
+ movzx r5d, word [r2 + 160] ; [16]900
+ mov [rsp], r4w901
+ mov [r2 - 2], r5w902
+903
+ lea r4, [r1 * 3]904
+ lea r5, [r0 + 32]905
+ call ang16_mode_11_25906
+ sub r2, 2907
+ call ang16_mode_11_25908
+ add r2, 34909
+ mov r0, r5910
+ call ang16_mode_11_25911
+ sub r2, 2912
+ call ang16_mode_11_25913
+ mov r5d, [rsp]914
+ mov [r2 - 32], r5w915
+ RET916
+917
+cglobal intra_pred_ang16_9, 3,7,17918
+ add r2, 64919
+ xor r6d, r6d920
+ lea r3, [ang_table_avx2 + 16 * 32]921
+ shl r1d, 1922
+ lea r4, [r1 * 3]923
+ vbroadcasti32x8 m15, [pd_16]924
+ call ang16_mode_9_27925
+ RET926
+927
+cglobal intra_pred_ang16_27, 3,7,17928
+ xor r6d, r6d929
+ inc r6d930
+ lea r3, [ang_table_avx2 + 16 * 32]931
+ shl r1d, 1932
+ lea r4, [r1 * 3]933
+ vbroadcasti32x8 m15, [pd_16]934
+ call ang16_mode_9_27935
+ RET936
+937
+cglobal intra_pred_ang16_11, 3,7,17, 0-4938
+ movzx r5d, word [r2 + 64]939
+ movzx r6d, word [r2]940
+ mov [rsp], r5w941
+ mov [r2 + 64], r6w942
+ vbroadcasti32x8 m15, [pd_16]943
+ add r2, 64944
+ xor r6d, r6d945
+ lea r3, [ang_table_avx2 + 16 * 32]946
+ shl r1d, 1947
+ lea r4, [r1 * 3]948
+ call ang16_mode_11_25949
+ mov r6d, [rsp]950
+ mov [r2], r6w951
+ RET952
+953
+cglobal intra_pred_ang16_25, 3,7,17954
+ xor r6d, r6d955
+ inc r6d956
+ vbroadcasti32x8 m15, [pd_16]957
+ lea r3, [ang_table_avx2 + 16 * 32]958
+ shl r1d, 1959
+ lea r4, [r1 * 3]960
+ call ang16_mode_11_25961
+ RET962
+cglobal ang16_mode_5_31963
+ test r6d, r6d964
+965
+ vbroadcasti32x8 m0, [r2 + 2] ; [16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1]966
+ vbroadcasti32x8 m1, [r2 + 4] ; [17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2]967
+968
+ punpcklwd m3, m0, m1 ; [13 12 12 11 11 10 10 9 5 4 4 3 3 2 2 1]969
+ punpckhwd m0, m1 ; [17 16 16 15 15 14 14 13 9 8 8 7 7 6 6 5]970
+971
+ vbroadcasti32x8 m1, [r2 + 18] ; [24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9]972
+ vbroadcasti32x8 m4, [r2 + 20] ; [25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10]973
+ punpcklwd m2, m1, m4 ; [21 20 20 19 19 18 18 17 13 12 12 11 11 10 10 9]974
+ punpckhwd m1, m4 ; [25 24 24 23 23 22 22 21 17 16 16 15 15 14 14 13]975
+976
+ pmaddwd m4, m3, [r3 + 1 * 32] ; [17]977
+ paddd m4, m15978
+ psrld m4, 5979
+ pmaddwd m5, m0, [r3 + 1 * 32]980
+ paddd m5, m15981
+ psrld m5, 5982
+ packusdw m4, m5983
+984
+ movu ym16, [r3 - 14 * 32] ; [2]985
+ vinserti32x8 m16, [r3 + 3 * 32] ,1 ; [19]986
+ palignr m6, m0, m3, 4987
+ pmaddwd m5, m6, m16988
+ paddd m5, m15989
+ psrld m5, 5990
+ palignr m7, m2, m0, 4991
+ pmaddwd m8, m7, m16992
+ paddd m8, m15993
+ psrld m8, 5994
+ packusdw m5, m8995
+ vextracti32x8 ym6, m5, 1996
+997
+ palignr m8, m0, m3, 8998
+ palignr m9, m2, m0, 8999
+ movu ym16, [r3 - 12 * 32] ; [4]1000
+ vinserti32x8 m16, [r3 + 5 * 32],1 ; [21]1001
+ pmaddwd m7, m8, m161002
+ paddd m7, m151003
+ psrld m7, 51004
+ pmaddwd m10, m9,m161005
+ paddd m10, m151006
+ psrld m10, 51007
+ packusdw m7, m101008
+ vextracti32x8 ym8, m7, 11009
+1010
+ palignr m10, m0, m3, 121011
+ palignr m11, m2, m0, 121012
+ movu ym16,[r3 - 10 * 32] ; [6]1013
+ vinserti32x8 m16, [r3 + 7 * 32] ,1 ; [23]1014
+ pmaddwd m9, m10, m161015
+ paddd m9, m151016
+ psrld m9, 51017
+ pmaddwd m3, m11, m161018
+ paddd m3, m151019
+ psrld m3, 51020
+ packusdw m9, m31021
+ vextracti32x8 ym10, m9, 11022
+1023
+ pmaddwd m11, m0, [r3 - 8 * 32] ; [8]1024
+ paddd m11, m151025
+ psrld m11, 51026
+ pmaddwd m3, m2, [r3 - 8 * 32]1027
+ paddd m3, m151028
+ psrld m3, 51029
+ packusdw m11, m31030
+1031
+ TRANSPOSE_STORE_AVX2 4, 5, 6, 7, 8, 9, 10, 11, 12, 3, 01032
+1033
+ pmaddwd m4, m0, [r3 + 9 * 32] ; [25]1034
+ paddd m4, m151035
+ psrld m4, 51036
+ pmaddwd m5, m2, [r3 + 9 * 32]1037
+ paddd m5, m151038
+ psrld m5, 51039
+ packusdw m4, m51040
+1041
+ palignr m6, m2, m0, 41042
+ movu ym16, [r3 - 6 * 32] ; [10]1043
+ vinserti32x8 m16, [r3 + 11 * 32] ,1 ; [27]1044
+ pmaddwd m5, m6,m161045
+ paddd m5, m151046
+ psrld m5, 51047
+ palignr m7, m1, m2, 41048
+ pmaddwd m3, m7,m161049
+ paddd m3, m151050
+ psrld m3, 51051
+ packusdw m5, m31052
+ vextracti32x8 ym6, m5, 11053
+1054
+ palignr m8, m2, m0, 81055
+ palignr m9, m1, m2, 81056
+ movu ym16, [r3 - 4 * 32] ; [12]1057
+ vinserti32x8 m16, [r3 + 13 * 32] ,1 ; [29]1058
+ pmaddwd m7, m8, m161059
+ paddd m7, m151060
+ psrld m7, 51061
+ pmaddwd m3, m9, m161062
+ paddd m3, m151063
+ psrld m3, 51064
+ packusdw m7, m31065
+ vextracti32x8 ym8, m7, 11066
+1067
+1068
+ palignr m10, m2, m0, 121069
+ palignr m11, m1, m2, 121070
+ movu ym16, [r3 - 2 * 32] ; [14]1071
+ vinserti32x8 m16, [r3 + 15 * 32],1 ; [31]1072
+ pmaddwd m9, m10, m161073
+ paddd m9, m151074
+ psrld m9, 51075
+ pmaddwd m3, m11, m161076
+ paddd m3, m151077
+ psrld m3, 51078
+ packusdw m9, m31079
+ vextracti32x8 ym10, m9, 11080
+1081
+ pmaddwd m2, [r3] ; [16]1082
+ paddd m2, m151083
+ psrld m2, 51084
+ pmaddwd m1, [r3]1085
+ paddd m1, m151086
+ psrld m1, 51087
+ packusdw m2, m11088
+ TRANSPOSE_STORE_AVX2 4, 5, 6, 7, 8, 9, 10, 2, 0, 1, 161089
+ ret1090
+;; angle 32, modes 5 and 311091
+cglobal ang32_mode_5_311092
+ test r6d, r6d1093
+1094
+ vbroadcasti32x8 m0, [r2 + 2] ; [16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1]1095
+ vbroadcasti32x8 m1, [r2 + 4] ; [17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2]1096
+1097
+ punpcklwd m3, m0, m1 ; [13 12 12 11 11 10 10 9 5 4 4 3 3 2 2 1]1098
+ punpckhwd m0, m1 ; [17 16 16 15 15 14 14 13 9 8 8 7 7 6 6 5]1099
+1100
+ vbroadcasti32x8 m1, [r2 + 18] ; [24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9]1101
+ vbroadcasti32x8 m4, [r2 + 20] ; [25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10]1102
+ punpcklwd m2, m1, m4 ; [21 20 20 19 19 18 18 17 13 12 12 11 11 10 10 9]1103
+ punpckhwd m1, m4 ; [25 24 24 23 23 22 22 21 17 16 16 15 15 14 14 13]1104
+1105
+ movu ym16, [r3 - 15 * 32] ; [1]1106
+ vinserti32x8 m16, [r3 + 2 * 32],1 ; [18]1107
+ pmaddwd m4, m3, m161108
+ paddd m4, m151109
+ psrld m4, 51110
+ pmaddwd m5, m0, m161111
+ paddd m5, m151112
+ psrld m5, 51113
+ packusdw m4, m51114
+ vextracti32x8 ym5, m4, 11115
+1116
+1117
+ palignr m7, m0, m3, 41118
+ movu ym16, [r3 - 13 * 32] ; [3]1119
+ vinserti32x8 m16, [r3 + 4 * 32] ,1 ; [20]1120
+ pmaddwd m6, m7, m161121
+ paddd m6, m151122
+ psrld m6, 51123
+ palignr m8, m2, m0, 41124
+ pmaddwd m9, m8,m161125
+ paddd m9, m151126
+ psrld m9, 51127
+ packusdw m6, m91128
+ vextracti32x8 ym7, m6, 11129
+1130
+1131
+ palignr m9, m0, m3, 81132
+ movu ym16, [r3 - 11 * 32] ; [5]1133
+ vinserti32x8 m16, [r3 + 6 * 32] ,1 ; [22]1134
+ pmaddwd m8, m9,m161135
+ paddd m8, m151136
+ psrld m8, 51137
+ palignr m10, m2, m0, 81138
+ pmaddwd m11, m10,m161139
+ paddd m11, m151140
+ psrld m11, 51141
+ packusdw m8, m111142
+ vextracti32x8 ym9, m8, 11143
+1144
+1145
+ palignr m11, m0, m3, 121146
+ movu ym16, [r3 - 9 * 32] ; [7]1147
+ vinserti32x8 m16, [r3 + 8 * 32] ,1 ; [24]1148
+ pmaddwd m10, m11,m161149
+ paddd m10, m151150
+ psrld m10, 51151
+ palignr m12, m2, m0, 121152
+ pmaddwd m3, m12, m161153
+ paddd m3, m151154
+ psrld m3, 51155
+ packusdw m10, m31156
+ vextracti32x8 ym11, m10, 11157
+1158
+1159
+1160
+ TRANSPOSE_STORE_AVX2 4, 5, 6, 7, 8, 9, 10, 11, 12, 3, 01161
+1162
+ movu ym16, [r3 - 7 * 32] ; [9]1163
+ vinserti32x8 m16, [r3 + 10 * 32] ,1 ; [26]1164
+ pmaddwd m4, m0, m161165
+ paddd m4, m151166
+ psrld m4, 51167
+ pmaddwd m5, m2, m161168
+ paddd m5, m151169
+ psrld m5, 51170
+ packusdw m4, m51171
+ vextracti32x8 ym5, m4, 11172
+1173
+1174
+ palignr m7, m2, m0, 41175
+ movu ym16, [r3 - 5 * 32] ; [11]1176
+ vinserti32x8 m16, [r3 + 12 * 32],1 ; [28]1177
+ pmaddwd m6, m7, m161178
+ paddd m6, m151179
+ psrld m6, 51180
+ palignr m8, m1, m2, 41181
+ pmaddwd m9, m8,m161182
+ paddd m9, m151183
+ psrld m9, 51184
+ packusdw m6, m91185
+ vextracti32x8 ym7, m6, 11186
+1187
+ palignr m9, m2, m0, 81188
+ movu ym16, [r3 - 3 * 32] ; [13]1189
+ vinserti32x8 m16, [r3 + 14 * 32] ,1 ; [30]1190
+ pmaddwd m8, m9, m161191
+ paddd m8, m151192
+ psrld m8, 51193
+ palignr m3, m1, m2, 81194
+ pmaddwd m10, m3, m161195
+ paddd m10, m151196
+ psrld m10, 51197
+ packusdw m8, m101198
+ vextracti32x8 ym9, m8, 11199
+1200
+1201
+1202
+ palignr m10, m2, m0, 121203
+ pmaddwd m10, [r3 - 1 * 32] ; [15]1204
+ paddd m10, m151205
+ psrld m10, 51206
+ palignr m11, m1, m2, 121207
+ pmaddwd m11, [r3 - 1 * 32]1208
+ paddd m11, m151209
+ psrld m11, 51210
+ packusdw m10, m111211
+1212
+ pmaddwd m2, [r3 - 16 * 32] ; [0]1213
+ paddd m2, m151214
+ psrld m2, 51215
+ pmaddwd m1, [r3 - 16 * 32]1216
+ paddd m1, m151217
+ psrld m1, 51218
+ packusdw m2, m11219
+ TRANSPOSE_STORE_AVX2 4, 5, 6, 7, 8, 9, 10, 2, 0, 1, 161220
+ ret1221
+cglobal intra_pred_ang32_5, 3,8,171222
+ add r2, 1281223
+ xor r6d, r6d1224
+ lea r3, [ang_table_avx2 + 16 * 32]1225
+ add r1d, r1d1226
+ lea r4, [r1 * 3]1227
+ lea r7, [r0 + 8 * r1]1228
+ vbroadcasti32x8 m15, [pd_16]1229
+ call ang16_mode_5_311230
+1231
+ add r2, 181232
+ lea r0, [r0 + 32]1233
+1234
+ call ang32_mode_5_311235
+1236
+ add r2, 141237
+ lea r0, [r7 + 8 * r1]1238
+1239
+ call ang16_mode_5_311240
+ vbroadcasti32x8 m15, [pd_16]1241
+ add r2, 181242
+ lea r0, [r0 + 32]1243
+ call ang32_mode_5_311244
+ RET1245
+cglobal intra_pred_ang32_31, 3,7,171246
+ xor r6d, r6d1247
+ inc r6d1248
+ lea r3, [ang_table_avx2 + 16 * 32]1249
+ add r1d, r1d1250
+ lea r4, [r1 * 3]1251
+ lea r5, [r0 + 32]1252
+ vbroadcasti32x8 m15, [pd_16]1253
+ call ang16_mode_5_311254
+1255
+ add r2, 181256
+1257
+ call ang32_mode_5_311258
+1259
+ add r2, 141260
+ mov r0, r51261
+1262
+ call ang16_mode_5_311263
+1264
+ add r2, 181265
+ call ang32_mode_5_311266
+ RET1267
+cglobal intra_pred_ang16_5, 3,7,171268
+ add r2, 641269
+ xor r6d, r6d1270
+ vbroadcasti32x8 m15, [pd_16]1271
+ lea r3, [ang_table_avx2 + 16 * 32]1272
+ add r1d, r1d1273
+ lea r4, [r1 * 3]1274
+ call ang16_mode_5_311275
+ RET1276
+cglobal intra_pred_ang16_31, 3,7,171277
+ xor r6d, r6d1278
+ inc r6d1279
+ vbroadcasti32x8 m15, [pd_16]1280
+ lea r3, [ang_table_avx2 + 16 * 32]1281
+ add r1d, r1d1282
+ lea r4, [r1 * 3]1283
+ call ang16_mode_5_311284
+ RET1285
+;; angle 16, modes 4 and 321286
+cglobal ang16_mode_4_321287
+ test r6d, r6d1288
+1289
+ vbroadcasti32x8 m0, [r2 + 2] ; [16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1]1290
+ vbroadcasti32x8 m1, [r2 + 4] ; [17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2]1291
+1292
+ punpcklwd m3, m0, m1 ; [13 12 12 11 11 10 10 9 5 4 4 3 3 2 2 1]1293
+ punpckhwd m0, m1 ; [17 16 16 15 15 14 14 13 9 8 8 7 7 6 6 5]1294
+1295
+ vbroadcasti32x8 m1, [r2 + 18] ; [24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9]1296
+ vbroadcasti32x8 m4, [r2 + 20] ; [25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10]1297
+ punpcklwd m2, m1, m4 ; [21 20 20 19 19 18 18 17 13 12 12 11 11 10 10 9]1298
+ punpckhwd m1, m4 ; [25 24 24 23 23 22 22 21 17 16 16 15 15 14 14 13]1299
+1300
+ pmaddwd m4, m3, [r3 + 3 * 32] ; [21]1301
+ paddd m4, m151302
+ psrld m4, 51303
+ pmaddwd m5, m0, [r3 + 3 * 32]1304
+ paddd m5, m151305
+ psrld m5, 51306
+ packusdw m4, m51307
+1308
+ palignr m6, m0, m3, 4 ; [14 13 13 12 12 11 11 10 6 5 5 4 4 3 3 2]1309
+ palignr m7, m2, m0, 4 ; [18 17 17 16 16 15 15 14 10 9 9 8 8 7 7 6]1310
+ movu ym16,[r3 - 8 * 32] ; [10]1311
+ vinserti32x8 m16, [r3 + 13 * 32] ,1 ; [31]1312
+ pmaddwd m5, m6, m161313
+ paddd m5, m151314
+ psrld m5, 51315
+ pmaddwd m8, m7,m161316
+ paddd m8, m151317
+ psrld m8, 51318
+ packusdw m5, m81319
+ vextracti32x8 ym6, m5, 11320
+1321
+1322
+ palignr m7, m0, m3, 8 ; [15 14 14 13 13 12 12 11 7 6 6 5 5 4 4 3]1323
+ pmaddwd m7, [r3 + 2 * 32] ; [20]1324
+ paddd m7, m151325
+ psrld m7, 51326
+ palignr m8, m2, m0, 8 ; [19 18 18 17 17 16 16 15 11 10 10 9 9 8 8 7]1327
+ pmaddwd m8, [r3 + 2 * 32]1328
+ paddd m8, m151329
+ psrld m8, 51330
+ packusdw m7, m81331
+1332
+ palignr m9, m0, m3, 121333
+ palignr m3, m2, m0, 121334
+ movu ym16,[r3 - 9 * 32] ; [9]1335
+ vinserti32x8 m16, [r3 + 12 * 32] ,1 ; [30]1336
+ pmaddwd m8, m9, m161337
+ paddd m8, m151338
+ psrld m8, 51339
+ pmaddwd m10, m3,m161340
+ paddd m10,m151341
+ psrld m10, 51342
+ packusdw m8, m101343
+ vextracti32x8 ym9, m8, 11344
+1345
+1346
+ pmaddwd m10, m0, [r3 + 1 * 32] ; [19]1347
+ paddd m10,m151348
+ psrld m10, 51349
+ pmaddwd m3, m2, [r3 + 1 * 32]1350
+ paddd m3, m151351
+ psrld m3, 51352
+ packusdw m10, m31353
+1354
+ palignr m11, m2, m0, 41355
+ pmaddwd m11, [r3 - 10 * 32] ; [8]1356
+ paddd m11, m151357
+ psrld m11, 51358
+ palignr m3, m1, m2, 41359
+ pmaddwd m3, [r3 - 10 * 32]1360
+ paddd m3, m151361
+ psrld m3, 51362
+ packusdw m11, m31363
+1364
+ TRANSPOSE_STORE_AVX2 4, 5, 6, 7, 8, 9, 10, 11, 12, 3, 01365
+1366
+ palignr m4, m2, m0, 41367
+ pmaddwd m4, [r3 + 11 * 32] ; [29]1368
+ paddd m4, m151369
+ psrld m4, 51370
+ palignr m5, m1, m2, 41371
+ pmaddwd m5, [r3 + 11 * 32]1372
+ paddd m5, m151373
+ psrld m5, 51374
+ packusdw m4, m51375
+1376
+ palignr m5, m2, m0, 81377
+ pmaddwd m5, [r3] ; [18]1378
+ paddd m5, m151379
+ psrld m5, 51380
+ palignr m6, m1, m2, 81381
+ pmaddwd m6, [r3]1382
+ paddd m6, m151383
+ psrld m6, 51384
+ packusdw m5, m61385
1386
+ palignr m7, m2, m0, 121387
+ palignr m8, m1, m2, 121388
+ movu ym16,[r3 - 11 * 32] ; [7]1389
+ vinserti32x8 m16, [r3 + 10 * 32],1 ; [28]1390
+ pmaddwd m6, m7, m161391
+ paddd m6, m151392
+ psrld m6, 51393
+ palignr m8, m1, m2, 121394
+ pmaddwd m3, m8, m161395
+ paddd m3,m151396
+ psrld m3, 51397
+ packusdw m6, m31398
+ vextracti32x8 ym7, m6, 11399
+1400
+ movu m0, [r2 + 34] ; [32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17]1401
+ pmaddwd m8, m2, [r3 - 1 * 32] ; [17]1402
+ paddd m8, m151403
+ psrld m8, 51404
+ pmaddwd m9, m1, [r3 - 1 * 32]1405
+ paddd m9, m151406
+ psrld m9, 51407
+ packusdw m8, m91408
+1409
+ palignr m3, m0, m0, 2 ; [ x 32 31 30 29 28 27 26 x 24 23 22 21 20 19 18]1410
+ punpcklwd m0, m3 ; [29 29 28 28 27 27 26 22 21 20 20 19 19 18 18 17]1411
+1412
+ palignr m10, m1, m2, 41413
+ pmaddwd m9, m10, [r3 - 12 * 32] ; [6]1414
+ paddd m9, m151415
+ psrld m9, 51416
+ palignr m11, m0, m1, 41417
+ pmaddwd m3, m11, [r3 - 12 * 32]1418
+ paddd m3, m151419
+ psrld m3, 51420
+ packusdw m9, m31421
+1422
+ pmaddwd m10, [r3 + 9 * 32] ; [27]1423
+ paddd m10,m151424
+ psrld m10, 51425
+ pmaddwd m11, [r3 + 9 * 32]1426
+ paddd m11, m151427
+ psrld m11, 51428
+ packusdw m10, m111429
+1430
+ palignr m3, m1, m2, 81431
+ pmaddwd m3, [r3 - 2 * 32] ; [16]1432
+ paddd m3, m151433
+ psrld m3, 51434
+ palignr m0, m1, 81435
+ pmaddwd m0, [r3 - 2 * 32]1436
+ paddd m0,m151437
+ psrld m0, 51438
+ packusdw m3, m01439
+ TRANSPOSE_STORE_AVX2 4, 5, 6, 7, 8, 9, 10, 3, 0, 1, 161440
+ ret1441
+;; angle 32, modes 4 and 321442
+cglobal ang32_mode_4_321443
+ test r6d, r6d1444
+1445
+ vbroadcasti32x8 m0, [r2 + 2] ; [16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1]1446
+ vbroadcasti32x8 m1, [r2 + 4] ; [17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2]1447
+1448
+ punpcklwd m3, m0, m1 ; [13 12 12 11 11 10 10 9 5 4 4 3 3 2 2 1]1449
+ punpckhwd m0, m1 ; [17 16 16 15 15 14 14 13 9 8 8 7 7 6 6 5]1450
+1451
+ vbroadcasti32x8 m1, [r2 + 18] ; [24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9]1452
+ vbroadcasti32x8 m4, [r2 + 20] ; [25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10]1453
+ punpcklwd m2, m1, m4 ; [21 20 20 19 19 18 18 17 13 12 12 11 11 10 10 9]1454
+ punpckhwd m1, m4 ; [25 24 24 23 23 22 22 21 17 16 16 15 15 14 14 13]1455
+1456
+ movu ym16, [r3 - 13 * 32] ; [5]1457
+ vinserti32x8 m16, [r3 + 8 * 32],1 ; [26]1458
+ pmaddwd m4, m3, m161459
+ paddd m4, m151460
+ psrld m4, 51461
+ pmaddwd m5, m0,m161462
+ paddd m5, m151463
+ psrld m5, 51464
+ packusdw m4, m51465
+ vextracti32x8 ym5, m4, 11466
+1467
+ palignr m6, m0, m3, 4 ; [14 13 13 12 12 11 11 10 6 5 5 4 4 3 3 2]1468
+ pmaddwd m6, [r3 - 3 * 32] ; [15]1469
+ paddd m6, m151470
+ psrld m6, 51471
+ palignr m7, m2, m0, 4 ; [18 17 17 16 16 15 15 14 10 9 9 8 8 7 7 6]1472
+ pmaddwd m7, [r3 - 3 * 32]1473
+ paddd m7, m151474
+ psrld m7, 51475
+ packusdw m6, m71476
+1477
+ palignr m8, m0, m3, 8 ; [15 14 14 13 13 12 12 11 7 6 6 5 5 4 4 3]1478
+ palignr m9, m2, m0, 8 ; [19 18 18 17 17 16 16 15 11 10 10 9 9 8 8 7]1479
+ movu ym16, [r3 - 14 * 32] ; [4]1480
+ vinserti32x8 m16, [r3 + 7 * 32] ,1 ; [25]1481
+ pmaddwd m7, m8, m161482
+ paddd m7, m151483
+ psrld m7, 51484
+ pmaddwd m10, m9, m161485
+ paddd m10, m151486
+ psrld m10, 51487
+ packusdw m7, m101488
+ vextracti32x8 ym8, m7, 11489
+1490
+ palignr m9, m0, m3, 121491
+ pmaddwd m9, [r3 - 4 * 32] ; [14]1492
+ paddd m9, m151493
+ psrld m9, 51494
+ palignr m3, m2, m0, 121495
+ pmaddwd m3, [r3 - 4 * 32]1496
+ paddd m3,m151497
+ psrld m3, 51498
+ packusdw m9, m31499
+1500
+ movu ym16, [r3 - 15 * 32] ; [3]1501
+ vinserti32x8 m16, [r3 + 6 * 32] ,1 ; [24]1502
+ pmaddwd m10, m0, m161503
+ paddd m10, m151504
+ psrld m10, 51505
+ pmaddwd m3, m2, m161506
+ paddd m3,m151507
+ psrld m3, 51508
+ packusdw m10, m31509
+ vextracti32x8 ym11, m10, 11510
+1511
+ TRANSPOSE_STORE_AVX2 4, 5, 6, 7, 8, 9, 10, 11, 12, 3, 01512
+1513
+ palignr m4, m2, m0, 41514
+ pmaddwd m4, [r3 - 5* 32] ; [13]1515
+ paddd m4, m151516
+ psrld m4, 51517
+ palignr m5, m1, m2, 41518
+ pmaddwd m5, [r3 - 5 * 32]1519
+ paddd m5, m151520
+ psrld m5, 51521
+ packusdw m4, m51522
+1523
+ palignr m6, m2, m0, 81524
+ palignr m7, m1, m2, 81525
+ movu ym16, [r3 - 16 * 32] ; [2]1526
+ vinserti32x8 m16, [r3 + 5 * 32] ,1 ; [23]1527
+ pmaddwd m5, m6, m161528
+ paddd m5, m151529
+ psrld m5, 51530
+ palignr m7, m1, m2, 81531
+ pmaddwd m8, m7,m161532
+ paddd m8, m151533
+ psrld m8, 51534
+ packusdw m5, m81535
+ vextracti32x8 ym6, m5, 11536
+1537
+1538
+ palignr m7, m2, m0, 121539
+ pmaddwd m7, [r3 - 6 * 32] ; [12]1540
+ paddd m7, m151541
+ psrld m7, 51542
+ palignr m8, m1, m2, 121543
+ pmaddwd m8, [r3 - 6 * 32]1544
+ paddd m8, m151545
+ psrld m8, 51546
+ packusdw m7, m81547
+1548
+ movu m0, [r2 + 34] ; [32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17]1549
+ pmaddwd m8, m2, [r3 - 17 * 32] ; [1]1550
+ paddd m8, m151551
+ psrld m8, 51552
+ pmaddwd m9, m1, [r3 - 17 * 32]1553
+ paddd m9, m151554
+ psrld m9, 51555
+ packusdw m8, m91556
+1557
+ palignr m3, m0, m0, 2 ; [ x 32 31 30 29 28 27 26 x 24 23 22 21 20 19 18]1558
+ punpcklwd m0, m3 ; [29 29 28 28 27 27 26 22 21 20 20 19 19 18 18 17]1559
+1560
+ pmaddwd m9, m2, [r3 + 4 * 32] ; [22]1561
+ paddd m9, m151562
+ psrld m9, 51563
+ pmaddwd m3, m1, [r3 + 4 * 32]1564
+ paddd m3, m151565
+ psrld m3, 51566
+ packusdw m9, m31567
+1568
+ palignr m10, m1, m2, 41569
+ pmaddwd m10, [r3 - 7 * 32] ; [11]1570
+ paddd m10, m151571
+ psrld m10, 51572
+ palignr m11, m0, m1, 41573
+ pmaddwd m11, [r3 - 7 * 32]1574
+ paddd m11, m151575
+ psrld m11, 51576
+ packusdw m10, m111577
+1578
+ palignr m3, m1, m2, 81579
+ pmaddwd m3, [r3 - 18 * 32] ; [0]1580
+ paddd m3, m151581
+ psrld m3, 51582
+ palignr m0, m1, 81583
+ pmaddwd m0, [r3 - 18 * 32]1584
+ paddd m0, m151585
+ psrld m0, 51586
+ packusdw m3, m01587
+ TRANSPOSE_STORE_AVX2 4, 5, 6, 7, 8, 9, 10, 3, 0, 1, 161588
+ ret1589
+cglobal intra_pred_ang32_4, 3,8,171590
+ add r2, 1281591
+ xor r6d, r6d1592
+ lea r3, [ang_table_avx2 + 18 * 32]1593
+ add r1d, r1d1594
+ lea r4, [r1 * 3]1595
+ lea r7, [r0 + 8 * r1]1596
+ vbroadcasti32x8 m15, [pd_16]1597
+ call ang16_mode_4_321598
+1599
+ add r2, 221600
+ lea r0, [r0 + 32]1601
+1602
+ call ang32_mode_4_321603
+1604
+ add r2, 101605
+ lea r0, [r7 + 8 * r1]1606
+1607
+ call ang16_mode_4_321608
+1609
+ add r2, 221610
+ lea r0, [r0 + 32]1611
+ call ang32_mode_4_321612
+ RET1613
+cglobal intra_pred_ang32_32, 3,7,171614
+ xor r6d, r6d1615
+ inc r6d1616
+ lea r3, [ang_table_avx2 + 18 * 32]1617
+ add r1d, r1d1618
+ lea r4, [r1 * 3]1619
+ lea r5, [r0 + 32]1620
+ vbroadcasti32x8 m15, [pd_16]1621
+ call ang16_mode_4_321622
+1623
+ add r2, 221624
+1625
+ call ang32_mode_4_321626
+1627
+ add r2, 101628
+ mov r0, r51629
+1630
+ call ang16_mode_4_321631
+ add r2, 221632
+ call ang32_mode_4_321633
+ RET1634
+cglobal intra_pred_ang16_4, 3,7,171635
+ add r2, 641636
+ xor r6d, r6d1637
+ vbroadcasti32x8 m15, [pd_16]1638
+ lea r3, [ang_table_avx2 + 18 * 32]1639
+ add r1d, r1d1640
+ lea r4, [r1 * 3]1641
+ call ang16_mode_4_321642
+ RET1643
+cglobal intra_pred_ang16_32, 3,7,171644
+ xor r6d, r6d1645
+ inc r6d1646
+ vbroadcasti32x8 m15, [pd_16]1647
+ lea r3, [ang_table_avx2 + 18 * 32]1648
+ shl r1d, 11649
+ lea r4, [r1 * 3]1650
+ call ang16_mode_4_321651
+ RET1652
+;; angle 16, modes 6 and 301653
+cglobal ang16_mode_6_301654
+ test r6d, r6d1655
+1656
+ vbroadcasti32x8 m0, [r2 + 2] ; [16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1]1657
+ vbroadcasti32x8 m1, [r2 + 4] ; [17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2]1658
+1659
+ punpcklwd m3, m0, m1 ; [13 12 12 11 11 10 10 9 5 4 4 3 3 2 2 1]1660
+ punpckhwd m0, m1 ; [17 16 16 15 15 14 14 13 9 8 8 7 7 6 6 5]1661
+1662
+ vbroadcasti32x8 m1, [r2 + 18] ; [24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9]1663
+ vbroadcasti32x8 m4, [r2 + 20] ; [25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10]1664
+ punpcklwd m2, m1, m4 ; [21 20 20 19 19 18 18 17 13 12 12 11 11 10 10 9]1665
+ punpckhwd m1, m4 ; [25 24 24 23 23 22 22 21 17 16 16 15 15 14 14 13]1666
+1667
+ movu ym16, [r3 - 2 * 32] ; [13]1668
+ vinserti32x8 m16, [r3 + 11 * 32] ,1 ; [26]1669
+ pmaddwd m4, m3, m161670
+ paddd m4, m151671
+ psrld m4, 51672
+ pmaddwd m5, m0, m161673
+ paddd m5, m151674
+ psrld m5, 51675
+ packusdw m4, m51676
+ vextracti32x8 ym5, m4, 11677
+1678
+ palignr m7, m0, m3, 41679
+ palignr m8, m2, m0, 41680
+ movu ym16, [r3 - 8 * 32] ; [7]1681
+ vinserti32x8 m16, [r3 + 5 * 32] ,1 ; [20]1682
+ pmaddwd m6, m7, m161683
+ paddd m6, m151684
+ psrld m6, 51685
+ pmaddwd m9, m8, m161686
+ paddd m9, m151687
+ psrld m9, 51688
+ packusdw m6, m91689
+ vextracti32x8 ym7, m6, 11690
+1691
+ palignr m10, m0, m3, 81692
+ palignr m11, m2, m0, 81693
+ movu ym16, [r3 - 14 * 32] ; [1]1694
+ vinserti32x8 m16, [r3 - 1 * 32],1 ; [14]1695
+ pmaddwd m8, m10, m161696
+ paddd m8,m151697
+ psrld m8, 51698
+ palignr m11, m2, m0, 81699
+ pmaddwd m9, m11, m161700
+ paddd m9, m151701
+ psrld m9, 51702
+ packusdw m8, m91703
+ vextracti32x8 ym9, m8, 11704
+1705
+ pmaddwd m10, [r3 + 12 * 32] ; [27]1706
+ paddd m10,m151707
+ psrld m10, 51708
+ pmaddwd m11, [r3 + 12 * 32]1709
+ paddd m11, m151710
+ psrld m11, 51711
+ packusdw m10, m111712
+1713
+ palignr m11, m0, m3, 121714
+ pmaddwd m11, [r3 - 7 * 32] ; [8]1715
+ paddd m11, m151716
+ psrld m11, 51717
+ palignr m12, m2, m0, 121718
+ pmaddwd m12, [r3 - 7 * 32]1719
+ paddd m12, m151720
+ psrld m12, 51721
+ packusdw m11, m121722
+1723
+ TRANSPOSE_STORE_AVX2 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 01724
+1725
+ palignr m4, m0, m3, 121726
+ pmaddwd m4, [r3 + 6 * 32] ; [21]1727
+ paddd m4, m151728
+ psrld m4, 51729
+ palignr m5, m2, m0, 121730
+ pmaddwd m5, [r3 + 6 * 32]1731
+ paddd m5, m151732
+ psrld m5, 51733
+ packusdw m4, m51734
+1735
+ movu ym16, [r3 - 13 * 32] ; [2]1736
+ vinserti32x8 m16, [r3] ,1 ; [15]1737
+ pmaddwd m5, m0, m161738
+ paddd m5, m151739
+ psrld m5, 51740
+ pmaddwd m3, m2,m161741
+ paddd m3, m151742
+ psrld m3, 51743
+ packusdw m5, m31744
+ vextracti32x8 ym6, m5, 11745
+1746
+ pmaddwd m7, m0, [r3 + 13 * 32] ; [28]1747
+ paddd m7, m151748
+ psrld m7, 51749
+ pmaddwd m3, m2, [r3 + 13 * 32]1750
+ paddd m3, m151751
+ psrld m3, 51752
+ packusdw m7, m31753
+1754
+ palignr m9, m2, m0, 41755
+ palignr m3, m1, m2, 41756
+ movu ym16, [r3 - 6 * 32] ; [9]1757
+ vinserti32x8 m16, [r3 + 7 * 32],1 ; [22]1758
+ pmaddwd m8, m9, m161759
+ paddd m8, m151760
+ psrld m8, 51761
+ pmaddwd m10, m3, m161762
+ paddd m10,m151763
+ psrld m10, 51764
+ packusdw m8, m101765
+ vextracti32x8 ym9, m8, 11766
+1767
+1768
+ palignr m11, m2, m0, 81769
+ pmaddwd m10, m11, [r3 - 12 * 32] ; [3]1770
+ paddd m10, m151771
+ psrld m10, 51772
+ palignr m3, m1, m2, 81773
+ pmaddwd m12, m3, [r3 - 12 * 32]1774
+ paddd m12, m151775
+ psrld m12, 51776
+ packusdw m10, m121777
+1778
+ pmaddwd m11, [r3 + 1 * 32] ; [16]1779
+ paddd m11, m151780
+ psrld m11, 51781
+ pmaddwd m3, [r3 + 1 * 32]1782
+ paddd m3, m151783
+ psrld m3, 51784
+ packusdw m11, m31785
+ TRANSPOSE_STORE_AVX2 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 161786
+ ret1787
+;; angle 32, modes 6 and 301788
+cglobal ang32_mode_6_301789
+ test r6d, r6d1790
+1791
+ vbroadcasti32x8 m0, [r2 + 2] ; [16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1]1792
+ vbroadcasti32x8 m1, [r2 + 4] ; [17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2]1793
+1794
+ punpcklwd m3, m0, m1 ; [13 12 12 11 11 10 10 9 5 4 4 3 3 2 2 1]1795
+ punpckhwd m0, m1 ; [17 16 16 15 15 14 14 13 9 8 8 7 7 6 6 5]1796
+1797
+ vbroadcasti32x8 m1, [r2 + 18] ; [24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9]1798
+ vbroadcasti32x8 m4, [r2 + 20] ; [25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10]1799
+ punpcklwd m2, m1, m4 ; [21 20 20 19 19 18 18 17 13 12 12 11 11 10 10 9]1800
+ punpckhwd m1, m4 ; [25 24 24 23 23 22 22 21 17 16 16 15 15 14 14 13]1801
+1802
+ pmaddwd m4, m3, [r3 + 14 * 32] ; [29]1803
+ paddd m4, m151804
+ psrld m4, 51805
+ pmaddwd m5, m0, [r3 + 14 * 32]1806
+ paddd m5, m151807
+ psrld m5, 51808
+ packusdw m4, m51809
+1810
+ palignr m6, m0, m3, 41811
+ palignr m7, m2, m0, 41812
+ movu ym16, [r3 - 5 * 32] ; [10]1813
+ vinserti32x8 m16, [r3 + 8 * 32] ,1 ; [23]1814
+ pmaddwd m5, m6, m161815
+ paddd m5, m151816
+ psrld m5, 51817
+ pmaddwd m8, m7, m161818
+ paddd m8, m151819
+ psrld m8, 51820
+ packusdw m5, m81821
+ vextracti32x8 ym6, m5, 11822
+1823
+ palignr m9, m0, m3, 81824
+ palignr m12, m2, m0, 81825
+ movu ym16, [r3 - 11 * 32] ; [4]1826
+ vinserti32x8 m16, [r3 + 2 * 32] ,1 ; [17]1827
+ pmaddwd m7, m9, m161828
+ paddd m7,m151829
+ psrld m7, 51830
+ palignr m12, m2, m0, 81831
+ pmaddwd m11, m12,m161832
+ paddd m11,m151833
+ psrld m11, 51834
+ packusdw m7, m111835
+ vextracti32x8 ym8, m7, 11836
+1837
+ pmaddwd m9, [r3 + 15 * 32] ; [30]1838
+ paddd m9, m151839
+ psrld m9, 51840
+ pmaddwd m12, [r3 + 15 * 32]1841
+ paddd m12, m151842
+ psrld m12, 51843
+ packusdw m9, m121844
+1845
+ palignr m11, m0, m3, 121846
+ pmaddwd m10, m11, [r3 - 4 * 32] ; [11]1847
+ paddd m10, m151848
+ psrld m10, 51849
+ palignr m12, m2, m0, 121850
+ pmaddwd m3, m12, [r3 - 4 * 32]1851
+ paddd m3, m151852
+ psrld m3, 51853
+ packusdw m10, m31854
+1855
+ pmaddwd m11, [r3 + 9 * 32] ; [24]1856
+ paddd m11, m151857
+ psrld m11, 51858
+ pmaddwd m12, [r3 + 9 * 32]1859
+ paddd m12,m151860
+ psrld m12, 51861
+ packusdw m11, m121862
+1863
+ TRANSPOSE_STORE_AVX2 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 01864
+1865
+ movu ym16, [r3 - 10 * 32] ; [5]1866
+ vinserti32x8 m16, [r3 + 3 * 32] ,1 ; [18]1867
+ pmaddwd m4, m0, m161868
+ paddd m4, m151869
+ psrld m4, 51870
+ pmaddwd m5, m2, m161871
+ paddd m5, m151872
+ psrld m5, 51873
+ packusdw m4, m51874
+ vextracti32x8 ym5, m4, 11875
+1876
+ pmaddwd m6, m0, [r3 + 16 * 32] ; [31]1877
+ paddd m6,m151878
+ psrld m6, 51879
+ pmaddwd m7, m2, [r3 + 16 * 32]1880
+ paddd m7,m151881
+ psrld m7, 51882
+ packusdw m6, m71883
+1884
+ palignr m8, m2, m0, 41885
+ palignr m9, m1, m2, 41886
+ movu ym16, [r3 - 3 * 32] ; [12]1887
+ vinserti32x8 m16, [r3 + 10 * 32],1 ; [25]1888
+ pmaddwd m7, m8,m161889
+ paddd m7,m151890
+ psrld m7, 51891
+ pmaddwd m3, m9, m161892
+ paddd m3, m151893
+ psrld m3, 51894
+ packusdw m7, m31895
+ vextracti32x8 ym8, m7, 11896
+1897
+ palignr m10, m2, m0, 81898
+ palignr m12, m1, m2, 81899
+ movu ym16, [r3 - 9 * 32] ; [6]1900
+ vinserti32x8 m16, [r3 + 4 * 32] ,1 ; [19]1901
+ pmaddwd m9, m10, m161902
+ paddd m9, m151903
+ psrld m9, 51904
+ pmaddwd m3, m12,m161905
+ paddd m3, m151906
+ psrld m3, 51907
+ packusdw m9, m31908
+ vextracti32x8 ym10, m9, 11909
+1910
+1911
+ palignr m11, m2, m0, 121912
+ pmaddwd m11, [r3 - 15 * 32] ; [0]1913
+ paddd m11, m151914
+ psrld m11, 51915
+ palignr m3, m1, m2, 121916
+ pmaddwd m3, [r3 - 15 * 32]1917
+ paddd m3, m151918
+ psrld m3, 51919
+ packusdw m11, m31920
+ TRANSPOSE_STORE_AVX2 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 161921
+ ret1922
+cglobal intra_pred_ang32_6, 3,8,171923
+ add r2, 1281924
+ xor r6d, r6d1925
+ lea r3, [ang_table_avx2 + 15 * 32]1926
+ add r1d, r1d1927
+ lea r4, [r1 * 3]1928
+ lea r7, [r0 + 8 * r1]1929
+ vbroadcasti32x8 m15, [pd_16]1930
+ call ang16_mode_6_301931
+1932
+ add r2, 121933
+ lea r0, [r0 + 32]1934
+1935
+ call ang32_mode_6_301936
+1937
+ add r2, 201938
+ lea r0, [r7 + 8 * r1]1939
+1940
+ call ang16_mode_6_301941
+1942
+ add r2, 121943
+ lea r0, [r0 + 32]1944
+ call ang32_mode_6_301945
+ RET1946
+cglobal intra_pred_ang32_30, 3,7,171947
+ xor r6d, r6d1948
+ inc r6d1949
+ lea r3, [ang_table_avx2 + 15 * 32]1950
+ add r1d, r1d1951
+ lea r4, [r1 * 3]1952
+ lea r5, [r0 + 32]1953
+ vbroadcasti32x8 m15, [pd_16]1954
+ call ang16_mode_6_301955
+1956
+ add r2, 121957
+1958
+ call ang32_mode_6_301959
+1960
+ add r2, 201961
+ mov r0, r51962
+1963
+ call ang16_mode_6_301964
+1965
+ add r2, 121966
+ call ang32_mode_6_301967
+ RET1968
+cglobal intra_pred_ang16_6, 3,7,171969
+ add r2, 641970
+ xor r6d, r6d1971
+ vbroadcasti32x8 m15, [pd_16]1972
+ lea r3, [ang_table_avx2 + 15 * 32]1973
+ shl r1d, 11974
+ lea r4, [r1 * 3]1975
+ call ang16_mode_6_301976
+ RET1977
+cglobal intra_pred_ang16_30, 3,7,171978
+ xor r6d, r6d1979
+ inc r6d1980
+ vbroadcasti32x8 m15, [pd_16]1981
+ lea r3, [ang_table_avx2 + 15 * 32]1982
+ shl r1d, 11983
+ lea r4, [r1 * 3]1984
+ call ang16_mode_6_301985
+ RET1986
+1987
+;; angle 16, modes 8 and 281988
+cglobal ang16_mode_8_281989
+ test r6d, r6d1990
+1991
+ vbroadcasti32x8 m0, [r2 + 2] ; [16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1]1992
+ vbroadcasti32x8 m1, [r2 + 4] ; [17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2]1993
+1994
+ punpcklwd m3, m0, m1 ; [13 12 12 11 11 10 10 9 5 4 4 3 3 2 2 1]1995
+ punpckhwd m0, m1 ; [17 16 16 15 15 14 14 13 9 8 8 7 7 6 6 5]1996
+1997
+ vbroadcasti32x8 m2, [r2 + 18] ; [24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9]1998
+ vbroadcasti32x8 m4, [r2 + 20] ; [25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10]1999
+ punpcklwd m2, m4 ; [21 20 20 19 19 18 18 17 13 12 12 11 11 10 10 9]2000
+2001
+ movu ym14, [r3 - 10 * 32]2002
+ vinserti32x8 m14, [r3 - 5 * 32], 12003
+ pmaddwd m4, m3, m14 ; [5], [10]2004
+ paddd m4, m152005
+ psrld m4, 52006
+ pmaddwd m5, m0, m142007
+ paddd m5, m152008
+ psrld m5, 52009
+ packusdw m4, m52010
+ vextracti32x8 ym5, m4, 12011
+2012
+ movu ym14, [r3]2013
+ vinserti32x8 m14, [r3 + 5 * 32], 12014
+ pmaddwd m6, m3, m14 ; [15], [20]2015
+ paddd m6, m152016
+ psrld m6, 52017
+ pmaddwd m9, m0, m142018
+ paddd m9, m152019
+ psrld m9, 52020
+ packusdw m6, m92021
+ vextracti32x8 ym7, m6, 12022
+2023
+ movu ym14, [r3 + 10 * 32]2024
+ vinserti32x8 m14, [r3 + 15 * 32], 12025
+ pmaddwd m8, m3, m14 ; [25], [30]2026
+ paddd m8, m152027
+ psrld m8, 52028
+ pmaddwd m9, m0, m142029
+ paddd m9, m152030
+ psrld m9, 52031
+ packusdw m8, m92032
+ vextracti32x8 ym9, m8, 12033
+2034
+ palignr m11, m0, m3, 42035
+ movu ym14, [r3 - 12 * 32]2036
+ vinserti32x8 m14, [r3 - 7 * 32], 12037
+ pmaddwd m10, m11, m14 ; [3], [8]2038
+ paddd m10, m152039
+ psrld m10, 52040
+ palignr m1, m2, m0, 42041
+ pmaddwd m12, m1, m142042
+ paddd m12, m152043
+ psrld m12, 52044
+ packusdw m10, m122045
+ vextracti32x8 ym11, m10, 12046
+2047
+ TRANSPOSE_STORE_AVX2 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 02048
+2049
+ palignr m7, m0, m3, 42050
+ movu ym14, [r3 - 2 * 32]2051
+ vinserti32x8 m14, [r3 + 3 * 32], 12052
+ pmaddwd m4, m7, m14 ; [13], [18]2053
+ paddd m4, m152054
+ psrld m4, 52055
+ palignr m1, m2, m0, 42056
+ pmaddwd m5, m1, m142057
+ paddd m5, m152058
+ psrld m5, 52059
+ packusdw m4, m52060
+ vextracti32x8 ym5, m4, 12061
+2062
+ movu ym14, [r3 + 8 * 32]2063
+ vinserti32x8 m14, [r3 + 13 * 32], 12064
+ pmaddwd m6, m7, m14 ; [23], [28]2065
+ paddd m6, m152066
+ psrld m6, 52067
+ pmaddwd m8, m1, m142068
+ paddd m8, m152069
+ psrld m8, 52070
+ packusdw m6, m82071
+ vextracti32x8 ym7, m6, 12072
+2073
+ movu ym14, [r3 - 14 * 32]2074
+ vinserti32x8 m14, [r3 - 9 * 32], 12075
+ palignr m1, m0, m3, 82076
+ pmaddwd m8, m1, m14 ; [1], [6]2077
+ paddd m8, m152078
+ psrld m8, 52079
+ palignr m2, m0, 82080
+ pmaddwd m9, m2, m142081
+ paddd m9, m152082
+ psrld m9, 52083
+ packusdw m8, m92084
+ vextracti32x8 ym9, m8, 12085
+2086
+ movu ym14, [r3 - 4 * 32]2087
+ vinserti32x8 m14, [r3 + 1 * 32], 12088
+ pmaddwd m3, m1, m14 ; [11], [16]2089
+ paddd m3, m152090
+ psrld m3, 52091
+ pmaddwd m0, m2, m142092
+ paddd m0, m152093
+ psrld m0, 52094
+ packusdw m3, m02095
+ vextracti32x8 ym1, m3, 12096
+ TRANSPOSE_STORE_AVX2 4, 5, 6, 7, 8, 9, 3, 1, 0, 2, 162097
+ ret2098
+2099
+;; angle 32, modes 8 and 282100
+cglobal ang32_mode_8_282101
+ test r6d, r6d2102
+2103
+ vbroadcasti32x8 m0, [r2 + 2] ; [16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1]2104
+ vbroadcasti32x8 m1, [r2 + 4] ; [17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2]2105
+2106
+ punpcklwd m3, m0, m1 ; [13 12 12 11 11 10 10 9 5 4 4 3 3 2 2 1]2107
+ punpckhwd m0, m1 ; [17 16 16 15 15 14 14 13 9 8 8 7 7 6 6 5]2108
+2109
+ vbroadcasti32x8 m2, [r2 + 18] ; [24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9]2110
+ vbroadcasti32x8 m4, [r2 + 20] ; [25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10]2111
+ punpcklwd m2, m4 ; [21 20 20 19 19 18 18 17 13 12 12 11 11 10 10 9]2112
+2113
+ movu ym14, [r3 + 6 * 32]2114
+ vinserti32x8 m14, [r3 + 11 * 32], 12115
+ pmaddwd m4, m3, m14 ; [21], [26]2116
+ paddd m4, m152117
+ psrld m4, 52118
+ pmaddwd m5, m0, m142119
+ paddd m5, m152120
+ psrld m5, 52121
+ packusdw m4, m52122
+ vextracti32x8 ym5, m4, 12123
+2124
+ pmaddwd m6, m3, [r3 + 16 * 32] ; [31]2125
+ paddd m6, [pd_16]2126
+ psrld m6, 52127
+ pmaddwd m9, m0, [r3 + 16 * 32]2128
+ paddd m9, [pd_16]2129
+ psrld m9, 52130
+ packusdw m6, m92131
+2132
+ palignr m11, m0, m3, 42133
+ movu ym14, [r3 - 11 * 32]2134
+ vinserti32x8 m14, [r3 - 6 * 32], 12135
+ pmaddwd m7, m11, m14 ; [4], [9]2136
+ paddd m7, m152137
+ psrld m7, 52138
+ palignr m1, m2, m0, 42139
+ pmaddwd m8, m1, m142140
+ paddd m8, m152141
+ psrld m8, 52142
+ packusdw m7, m82143
+ vextracti32x8 ym8, m7, 12144
+2145
+ movu ym14, [r3 - 1 * 32]2146
+ vinserti32x8 m14, [r3 + 4 * 32], 12147
+ pmaddwd m9, m11, m14 ; [14], [19]2148
+ paddd m9, m152149
+ psrld m9, 52150
+ pmaddwd m10, m1, m142151
+ paddd m10, m152152
+ psrld m10, 52153
+ packusdw m9, m102154
+ vextracti32x8 ym10, m9, 12155
+2156
+ pmaddwd m11, [r3 + 9 * 32] ; [24]2157
+ paddd m11, [pd_16]2158
+ psrld m11, 52159
+ pmaddwd m1, [r3 + 9 * 32]2160
+ paddd m1, [pd_16]2161
+ psrld m1, 52162
+ packusdw m11, m12163
+2164
+TRANSPOSE_STORE_AVX2 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 02165
+2166
+ palignr m4, m0, m3, 42167
+ pmaddwd m4, [r3 + 14 * 32] ; [29]2168
+ paddd m4, m152169
+ psrld m4, 52170
+ palignr m5, m2, m0, 42171
+ pmaddwd m5, [r3 + 14 * 32]2172
+ paddd m5, m152173
+ psrld m5, 52174
+ packusdw m4, m52175
+2176
+ palignr m1, m0, m3, 82177
+ pmaddwd m5, m1, [r3 - 13 * 32] ; [2]2178
+ paddd m5, m152179
+ psrld m5, 52180
+ palignr m10, m2, m0, 82181
+ pmaddwd m6, m10, [r3 - 13 * 32]2182
+ paddd m6, m152183
+ psrld m6, 52184
+ packusdw m5, m62185
+2186
+ movu ym14, [r3 - 8 * 32]2187
+ vinserti32x8 m14, [r3 - 3 * 32], 12188
+ pmaddwd m6, m1, m14 ; [7], [12]2189
+ paddd m6, m152190
+ psrld m6, 52191
+ pmaddwd m8, m10, m142192
+ paddd m8, m152193
+ psrld m8, 52194
+ packusdw m6, m82195
+ vextracti32x8 ym7, m6, 12196
+2197
+ movu ym14, [r3 + 2 * 32]2198
+ vinserti32x8 m14, [r3 + 7 * 32], 12199
+ pmaddwd m8, m1, m14 ; [17], [22]2200
+ paddd m8, m152201
+ psrld m8, 52202
+ pmaddwd m9, m10, m142203
+ paddd m9, m152204
+ psrld m9, 52205
+ packusdw m8, m92206
+ vextracti32x8 ym9, m8, 12207
+2208
+ pmaddwd m1, [r3 + 12 * 32] ; [27]2209
+ paddd m1, [pd_16]2210
+ psrld m1, 52211
+ pmaddwd m10, [r3 + 12 * 32]2212
+ paddd m10, [pd_16]2213
+ psrld m10, 52214
+ packusdw m1, m102215
+2216
+ palignr m11, m0, m3, 122217
+ pmaddwd m11, [r3 - 15 * 32] ; [0]2218
+ paddd m11, [pd_16]2219
+ psrld m11, 52220
+ palignr m2, m0, 122221
+ pmaddwd m2, [r3 - 15 * 32]2222
+ paddd m2, [pd_16]2223
+ psrld m2, 52224
+ packusdw m11, m22225
+ TRANSPOSE_STORE_AVX2 4, 5, 6, 7, 8, 9, 1, 11, 0, 2, 162226
+ ret2227
+2228
+2229
+cglobal intra_pred_ang32_8, 3,8,162230
+ add r2, 1282231
+ xor r6d, r6d2232
+ lea r3, [ang_table_avx2 + 15 * 32]2233
+ add r1d, r1d2234
+ lea r4, [r1 * 3]2235
+ lea r7, [r0 + 8 * r1]2236
+ vbroadcasti32x8 m15, [pd_16]2237
+2238
+ call ang16_mode_8_282239
+2240
+ add r2, 42241
+ lea r0, [r0 + 32]2242
+2243
+ call ang32_mode_8_282244
+2245
+ add r2, 282246
+ lea r0, [r7 + 8 * r1]2247
+2248
+ call ang16_mode_8_282249
+2250
+ add r2, 42251
+ lea r0, [r0 + 32]2252
+2253
+ call ang32_mode_8_282254
+ RET2255
+2256
+cglobal intra_pred_ang32_28, 3,7,162257
+ xor r6d, r6d2258
+ inc r6d2259
+ lea r3, [ang_table_avx2 + 15 * 32]2260
+ add r1d, r1d2261
+ lea r4, [r1 * 3]2262
+ lea r5, [r0 + 32]2263
+ vbroadcasti32x8 m15, [pd_16]2264
+ call ang16_mode_8_282265
+2266
+ add r2, 42267
+2268
+ call ang32_mode_8_282269
+2270
+ add r2, 282271
+ mov r0, r52272
+2273
+ call ang16_mode_8_282274
+2275
+ add r2, 42276
+ call ang32_mode_8_282277
+ RET2278
+2279
+ cglobal intra_pred_ang16_8, 3,7,162280
+ add r2, 642281
+ xor r6d, r6d2282
+ lea r3, [ang_table_avx2 + 15 * 32]2283
+ add r1d, r1d2284
+ lea r4, [r1 * 3]2285
+ vbroadcasti32x8 m15, [pd_16]2286
+2287
+ call ang16_mode_8_282288
+ RET2289
+2290
+cglobal intra_pred_ang16_28, 3,7,162291
+ xor r6d, r6d2292
+ inc r6d2293
+ lea r3, [ang_table_avx2 + 15 * 32]2294
+ add r1d, r1d2295
+ lea r4, [r1 * 3]2296
+ vbroadcasti32x8 m15, [pd_16]2297
+2298
+ call ang16_mode_8_282299
+ RET2300
+2301
+;; angle 16, modes 7 and 292302
+cglobal ang16_mode_7_292303
+ test r6d, r6d2304
+2305
+ vbroadcasti32x8 m0, [r2 + 2] ; [16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1]2306
+ vbroadcasti32x8 m1, [r2 + 4] ; [17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2]2307
+2308
+ punpcklwd m3, m0, m1 ; [13 12 12 11 11 10 10 9 5 4 4 3 3 2 2 1]2309
+ punpckhwd m0, m1 ; [17 16 16 15 15 14 14 13 9 8 8 7 7 6 6 5]2310
+2311
+ vbroadcasti32x8 m2, [r2 + 18] ; [24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9]2312
+ vbroadcasti32x8 m4, [r2 + 20] ; [25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10]2313
+ punpcklwd m2, m4 ; [21 20 20 19 19 18 18 17 13 12 12 11 11 10 10 9]2314
+2315
+ movu ym16, [r3 - 8 * 32] ; [9]2316
+ vinserti32x8 m16, [r3 + 1 * 32] ,1 ; [18]2317
+ pmaddwd m4, m3,m162318
+ paddd m4, m152319
+ psrld m4, 52320
+ pmaddwd m5, m0, m162321
+ paddd m5, m152322
+ psrld m5, 52323
+ packusdw m4, m52324
+ vextracti32x8 ym5, m4, 12325
+2326
+ pmaddwd m6, m3, [r3 + 10 * 32] ; [27]2327
+ paddd m6, m152328
+ psrld m6, 52329
+ pmaddwd m9, m0, [r3 + 10 * 32]2330
+ paddd m9, m152331
+ psrld m9, 52332
+ packusdw m6, m92333
+2334
+ palignr m10, m0, m3, 42335
+ pmaddwd m7, m10, [r3 - 13 * 32] ; [4]2336
+ paddd m7, m152337
+ psrld m7, 52338
+ palignr m11, m2, m0, 42339
+ pmaddwd m8, m11, [r3 - 13 * 32]2340
+ paddd m8, m152341
+ psrld m8, 52342
+ packusdw m7, m82343
+2344
+ movu ym16, [r3 - 4 * 32] ; [13]2345
+ vinserti32x8 m16, [r3 + 5 * 32],1 ; [22]2346
+ pmaddwd m8, m10, m162347
+ paddd m8, m152348
+ psrld m8, 52349
+ pmaddwd m9, m11, m162350
+ paddd m9, m152351
+ psrld m9, 52352
+ packusdw m8, m92353
+ vextracti32x8 ym9, m8, 12354
+2355
+ pmaddwd m10, [r3 + 14 * 32] ; [31]2356
+ paddd m10, m152357
+ psrld m10, 52358
+ pmaddwd m11, [r3 + 14 * 32]2359
+ paddd m11, m152360
+ psrld m11, 52361
+ packusdw m10, m112362
+2363
+ palignr m11, m0, m3, 82364
+ pmaddwd m11, [r3 - 9 * 32] ; [8]2365
+ paddd m11, m152366
+ psrld m11, 52367
+ palignr m12, m2, m0, 82368
+ pmaddwd m12, [r3 - 9 * 32]2369
+ paddd m12, m152370
+ psrld m12, 52371
+ packusdw m11, m122372
+2373
+ TRANSPOSE_STORE_AVX2 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 02374
+2375
+ palignr m5, m0, m3, 82376
+ palignr m6, m2, m0, 82377
+ movu ym16, [r3] ; [17]2378
+ vinserti32x8 m16, [r3 + 9 * 32] ,1 ; [26]2379
+ pmaddwd m4, m5, m162380
+ paddd m4, m152381
+ psrld m4, 52382
+ pmaddwd m7, m6, m162383
+ paddd m7, m152384
+ psrld m7, 52385
+ packusdw m4, m72386
+ vextracti32x8 ym5, m4, 12387
+2388
+2389
+ palignr m9, m0, m3, 122390
+ palignr m3, m2, m0, 122391
+ movu ym16, [r3 - 14 * 32] ; [3]2392
+ vinserti32x8 m16, [r3 - 5 * 32] ,1 ; [12]2393
+ pmaddwd m6, m9,m162394
+ paddd m6, m152395
+ psrld m6, 52396
+ pmaddwd m7, m3,m162397
+ paddd m7, m152398
+ psrld m7, 52399
+ packusdw m6, m72400
+ vextracti32x8 ym7, m6, 12401
+2402
+ movu ym16, [r3 + 4 * 32] ; [21]2403
+ vinserti32x8 m16, [r3 + 13 * 32] ,1 ; [30]2404
+ pmaddwd m8, m9,m162405
+ paddd m8, m152406
+ psrld m8, 52407
+ pmaddwd m10, m3, m162408
+ paddd m10, m152409
+ psrld m10, 52410
+ packusdw m8, m102411
+ vextracti32x8 ym9, m8, 12412
+2413
+ movu ym16,[r3 - 10 * 32] ; [7]2414
+ vinserti32x8 m16, [r3 - 1 * 32] ,1 ; [16]2415
+ pmaddwd m10, m0, m162416
+ paddd m10, m152417
+ psrld m10, 52418
+ pmaddwd m12, m2, m162419
+ paddd m12, m152420
+ psrld m12, 52421
+ packusdw m10, m122422
+ vextracti32x8 ym0, m10, 12423
+2424
+ TRANSPOSE_STORE_AVX2 4, 5, 6, 7, 8, 9, 10, 0, 1, 2, 162425
+ ret2426
+;; angle 32, modes 7 and 292427
+cglobal ang32_mode_7_292428
+ test r6d, r6d2429
+2430
+ vbroadcasti32x8 m0, [r2 + 2] ; [16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1]2431
+ vbroadcasti32x8 m1, [r2 + 4] ; [17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2]2432
+2433
+ punpcklwd m3, m0, m1 ; [13 12 12 11 11 10 10 9 5 4 4 3 3 2 2 1]2434
+ punpckhwd m0, m1 ; [17 16 16 15 15 14 14 13 9 8 8 7 7 6 6 5]2435
+2436
+ vbroadcasti32x8 m1, [r2 + 18] ; [24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9]2437
+ vbroadcasti32x8 m4, [r2 + 20] ; [25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10]2438
+ punpcklwd m2, m1, m4 ; [21 20 20 19 19 18 18 17 13 12 12 11 11 10 10 9]2439
+ punpckhwd m1, m4 ; [25 24 24 23 23 22 22 21 17 16 16 15 15 14 14 13]2440
+2441
+ pmaddwd m4, m3, [r3 + 8 * 32] ; [25]2442
+ paddd m4, m152443
+ psrld m4, 52444
+ pmaddwd m5, m0, [r3 + 8 * 32]2445
+ paddd m5, m152446
+ psrld m5, 52447
+ packusdw m4, m52448
+2449
+ palignr m8, m0, m3, 42450
+ pmaddwd m5, m8, [r3 - 15 * 32] ; [2]2451
+ paddd m5, m152452
+ psrld m5, 52453
+ palignr m9, m2, m0, 42454
+ pmaddwd m10, m9, [r3 - 15 * 32]2455
+ paddd m10, m152456
+ psrld m10, 52457
+ packusdw m5, m102458
+2459
+ movu ym16,[r3 - 6 * 32] ; [11]2460
+ vinserti32x8 m16, [r3 + 3 * 32],1 ; [20]2461
+ pmaddwd m6, m8, m162462
+ paddd m6, m152463
+ psrld m6, 52464
+ pmaddwd m7, m9, m162465
+ paddd m7, m152466
+ psrld m7, 52467
+ packusdw m6, m72468
+ vextracti32x8 ym7, m6, 12469
+2470
+ pmaddwd m8, [r3 + 12 * 32] ; [29]2471
+ paddd m8, m152472
+ psrld m8, 52473
+ pmaddwd m9, [r3 + 12 * 32]2474
+ paddd m9, m152475
+ psrld m9, 52476
+ packusdw m8, m92477
+2478
+ palignr m11, m0, m3, 82479
+ palignr m12, m2, m0, 82480
+ movu ym16, [r3 - 11 * 32] ; [6]2481
+ vinserti32x8 m16, [r3 - 2 * 32] ,1 ; [15]2482
+ pmaddwd m9, m11, m162483
+ paddd m9, m152484
+ psrld m9, 52485
+ palignr m12, m2, m0, 82486
+ pmaddwd m10, m12, m162487
+ paddd m10, m152488
+ psrld m10, 52489
+ packusdw m9, m102490
+ vextracti32x8 ym10, m9, 12491
+2492
+ pmaddwd m11, [r3 + 7 * 32] ; [24]2493
+ paddd m11, m152494
+ psrld m11, 52495
+ pmaddwd m12, [r3 + 7 * 32]2496
+ paddd m12, m152497
+ psrld m12, 52498
+ packusdw m11, m122499
+2500
+ TRANSPOSE_STORE_AVX2 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 02501
+2502
+ palignr m5, m0, m3, 122503
+ palignr m6, m2, m0, 122504
+ movu ym16, [r3 - 16 * 32] ; [1]2505
+ vinserti32x8 m16, [r3 - 7 * 32] ,1 ; [10]2506
+ pmaddwd m4, m5, m162507
+ paddd m4, m152508
+ psrld m4, 52509
+ pmaddwd m7, m6, m162510
+ paddd m7, m152511
+ psrld m7, 52512
+ packusdw m4, m72513
+ vextracti32x8 ym5, m4, 12514
+2515
+ palignr m9, m0, m3, 122516
+ pmaddwd m6, m9, [r3 + 2 * 32] ; [19]2517
+ paddd m6, m152518
+ psrld m6, 52519
+ palignr m3, m2, m0, 122520
+ pmaddwd m7, m3, [r3 + 2 * 32]2521
+ paddd m7, m152522
+ psrld m7, 52523
+ packusdw m6, m72524
+2525
+ pmaddwd m7, m9, [r3 + 11 * 32] ; [28]2526
+ paddd m7, m152527
+ psrld m7, 52528
+ pmaddwd m8, m3, [r3 + 11 * 32]2529
+ paddd m8, m152530
+ psrld m8, 52531
+ packusdw m7, m82532
+2533
+ movu ym16, [r3 - 12 * 32] ; [5]2534
+ vinserti32x8 m16, [r3 - 3 * 32] ,1 ; [14]2535
+ pmaddwd m8, m0, m162536
+ paddd m8, m152537
+ psrld m8, 52538
+ pmaddwd m10, m2, m162539
+ paddd m10,m152540
+ psrld m10, 52541
+ packusdw m8, m102542
+ vextracti32x8 ym9, m8, 12543
+2544
+ pmaddwd m10, m0, [r3 + 6 * 32] ; [23]2545
+ paddd m10,m152546
+ psrld m10, 52547
+ pmaddwd m12, m2, [r3 + 6 * 32]2548
+ paddd m12, m152549
+ psrld m12, 52550
+ packusdw m10, m122551
+2552
+ palignr m11, m2, m0, 42553
+ pmaddwd m11, [r3 - 17 * 32] ; [0]2554
+ paddd m11, m152555
+ psrld m11, 52556
+ palignr m12, m1, m2, 42557
+ pmaddwd m12, [r3 - 17 * 32]2558
+ paddd m12, m152559
+ psrld m12, 52560
+ packusdw m11, m122561
+ TRANSPOSE_STORE_AVX2 4, 5, 6, 7, 8, 9, 10, 11, 3, 2, 162562
+ ret2563
+2564
+cglobal intra_pred_ang32_7, 3,8,172565
+ add r2, 1282566
+ xor r6d, r6d2567
+ lea r3, [ang_table_avx2 + 17 * 32]2568
+ add r1d, r1d2569
+ lea r4, [r1 * 3]2570
+ lea r7, [r0 + 8 * r1]2571
+ vbroadcasti32x8 m15, [pd_16]2572
+ call ang16_mode_7_292573
+2574
+ add r2, 82575
+ lea r0, [r0 + 32]2576
+2577
+ call ang32_mode_7_292578
+2579
+ add r2, 242580
+ lea r0, [r7 + 8 * r1]2581
+2582
+ call ang16_mode_7_292583
+2584
+ add r2, 82585
+ lea r0, [r0 + 32]2586
+2587
+ call ang32_mode_7_292588
+ RET2589
+2590
+cglobal intra_pred_ang32_29, 3,7,172591
+ xor r6d, r6d2592
+ inc r6d2593
+ lea r3, [ang_table_avx2 + 17 * 32]2594
+ add r1d, r1d2595
+ lea r4, [r1 * 3]2596
+ lea r5, [r0 + 32]2597
+ vbroadcasti32x8 m15, [pd_16]2598
+ call ang16_mode_7_292599
+2600
+ add r2, 82601
+2602
+ call ang32_mode_7_292603
+2604
+ add r2, 242605
+ mov r0, r52606
+2607
+ call ang16_mode_7_292608
+ add r2, 82609
+ call ang32_mode_7_292610
+ RET2611
+cglobal intra_pred_ang16_7, 3,7,172612
+ add r2, 642613
+ xor r6d, r6d2614
+ vbroadcasti32x8 m15, [pd_16]2615
+ lea r3, [ang_table_avx2 + 17 * 32]2616
+ add r1d, r1d2617
+ lea r4, [r1 * 3]2618
+2619
+ call ang16_mode_7_292620
+ RET2621
+2622
+cglobal intra_pred_ang16_29, 3,7,172623
+ xor r6d, r6d2624
+ inc r6d2625
+ vbroadcasti32x8 m15, [pd_16]2626
+ lea r3, [ang_table_avx2 + 17 * 32]2627
+ add r1d, r1d2628
+ lea r4, [r1 * 3]2629
+2630
+ call ang16_mode_7_292631
+ RET2632
+;-------------------------------------------------------------------------------------------------------2633
+; avx512 code for intra_pred_ang32 mode 2 to 34 end2634
+;-------------------------------------------------------------------------------------------------------2635
%macro MODE_2_34 02636
movu m0, [r2 + 4]2637
movu m1, [r2 + 20]2638
x265_2.7.tar.gz/source/common/x86/ipfilter16.asm -> x265_2.9.tar.gz/source/common/x86/ipfilter16.asm
Changed
9510
1
2
%endif3
4
5
-SECTION_RODATA 326
+SECTION_RODATA 647
8
tab_c_524800: times 4 dd 5248009
tab_c_n8192: times 8 dw -819210
pd_524800: times 8 dd 52480011
12
+tab_ChromaCoeff: dw 0, 64, 0, 013
+ dw -2, 58, 10, -214
+ dw -4, 54, 16, -215
+ dw -6, 46, 28, -416
+ dw -4, 36, 36, -417
+ dw -4, 28, 46, -618
+ dw -2, 16, 54, -419
+ dw -2, 10, 58, -220
+ 21
+tab_LumaCoeff: dw 0, 0, 0, 64, 0, 0, 0, 022
+ dw -1, 4, -10, 58, 17, -5, 1, 023
+ dw -1, 4, -11, 40, 40, -11, 4, -124
+ dw 0, 1, -5, 17, 58, -10, 4, -125
+26
+ALIGN 6427
+tab_LumaCoeffH_avx512:28
+ times 4 dw 0, 0, 0, 64, 0, 0, 0, 029
+ times 4 dw -1, 4, -10, 58, 17, -5, 1, 030
+ times 4 dw -1, 4, -11, 40, 40, -11, 4, -131
+ times 4 dw 0, 1, -5, 17, 58, -10, 4, -132
+33
ALIGN 3234
tab_LumaCoeffV: times 4 dw 0, 035
times 4 dw 0, 6436
37
times 4 dw -5, 1738
times 4 dw 58, -1039
times 4 dw 4, -140
+41
ALIGN 3242
tab_LumaCoeffVer: times 8 dw 0, 043
times 8 dw 0, 6444
45
times 8 dw -5, 1746
times 8 dw 58, -1047
times 8 dw 4, -148
-49
+ 50
+ALIGN 6451
+const tab_ChromaCoeffV_avx512, times 16 dw 0, 6452
+ times 16 dw 0, 053
+54
+ times 16 dw -2, 5855
+ times 16 dw 10, -256
+57
+ times 16 dw -4, 5458
+ times 16 dw 16, -259
+60
+ times 16 dw -6, 4661
+ times 16 dw 28, -462
+63
+ times 16 dw -4, 3664
+ times 16 dw 36, -465
+66
+ times 16 dw -4, 2867
+ times 16 dw 46, -668
+69
+ times 16 dw -2, 1670
+ times 16 dw 54, -471
+72
+ times 16 dw -2, 1073
+ times 16 dw 58, -274
+75
+ALIGN 6476
+tab_LumaCoeffVer_avx512: times 16 dw 0, 077
+ times 16 dw 0, 6478
+ times 16 dw 0, 079
+ times 16 dw 0, 080
+81
+ times 16 dw -1, 482
+ times 16 dw -10, 5883
+ times 16 dw 17, -584
+ times 16 dw 1, 085
+86
+ times 16 dw -1, 487
+ times 16 dw -11, 4088
+ times 16 dw 40, -1189
+ times 16 dw 4, -190
+91
+ times 16 dw 0, 192
+ times 16 dw -5, 1793
+ times 16 dw 58, -1094
+ times 16 dw 4, -195
+96
+ALIGN 6497
+const interp8_hpp_shuf1_load_avx512, times 4 db 0, 1, 2, 3, 4, 5, 6, 7, 2, 3, 4, 5, 6, 7, 8, 998
+99
+ALIGN 64100
+const interp8_hpp_shuf2_load_avx512, times 4 db 4, 5, 6, 7, 8, 9, 10, 11, 6, 7, 8, 9, 10, 11, 12, 13101
+102
+ALIGN 64103
+const interp8_hpp_shuf1_store_avx512, times 4 db 0, 1, 4, 5, 2, 3, 6, 7, 8, 9, 12, 13, 10, 11, 14, 15104
+ 105
SECTION .text106
cextern pd_8107
cextern pd_32108
109
;-------------------------------------------------------------------------------------------------------------110
; void interp_8tap_vert_pp_%2x%3(pixel *src, intptr_t srcStride, pixel *dst, intptr_t dstStride, int coeffIdx)111
;-------------------------------------------------------------------------------------------------------------112
+%if ARCH_X86_64113
FILTER_VER_LUMA_sse2 pp, 4, 4114
FILTER_VER_LUMA_sse2 pp, 8, 8115
FILTER_VER_LUMA_sse2 pp, 8, 4116
117
FILTER_VER_LUMA_sse2 ps, 48, 64118
FILTER_VER_LUMA_sse2 ps, 64, 16119
FILTER_VER_LUMA_sse2 ps, 16, 64120
+%endif121
+122
+;-----------------------------------------------------------------------------123
+;p2s and p2s_aligned avx512 code start124
+;-----------------------------------------------------------------------------125
+%macro P2S_64x4_AVX512 0126
+ movu m0, [r0]127
+ movu m1, [r0 + r1]128
+ movu m2, [r0 + r1 * 2]129
+ movu m3, [r0 + r5]130
+ psllw m0, (14 - BIT_DEPTH)131
+ psllw m1, (14 - BIT_DEPTH)132
+ psllw m2, (14 - BIT_DEPTH)133
+ psllw m3, (14 - BIT_DEPTH)134
+ psubw m0, m4135
+ psubw m1, m4136
+ psubw m2, m4137
+ psubw m3, m4138
+ movu [r2], m0139
+ movu [r2 + r3], m1140
+ movu [r2 + r3 * 2], m2141
+ movu [r2 + r4], m3142
+143
+ movu m0, [r0 + mmsize]144
+ movu m1, [r0 + r1 + mmsize]145
+ movu m2, [r0 + r1 * 2 + mmsize]146
+ movu m3, [r0 + r5 + mmsize]147
+ psllw m0, (14 - BIT_DEPTH)148
+ psllw m1, (14 - BIT_DEPTH)149
+ psllw m2, (14 - BIT_DEPTH)150
+ psllw m3, (14 - BIT_DEPTH)151
+ psubw m0, m4152
+ psubw m1, m4153
+ psubw m2, m4154
+ psubw m3, m4155
+ movu [r2 + mmsize], m0156
+ movu [r2 + r3 + mmsize], m1157
+ movu [r2 + r3 * 2 + mmsize], m2158
+ movu [r2 + r4 + mmsize], m3159
+%endmacro160
+161
+%macro P2S_ALIGNED_64x4_AVX512 0162
+ mova m0, [r0]163
+ mova m1, [r0 + r1]164
+ mova m2, [r0 + r1 * 2]165
+ mova m3, [r0 + r5]166
+ psllw m0, (14 - BIT_DEPTH)167
+ psllw m1, (14 - BIT_DEPTH)168
+ psllw m2, (14 - BIT_DEPTH)169
+ psllw m3, (14 - BIT_DEPTH)170
+ psubw m0, m4171
+ psubw m1, m4172
+ psubw m2, m4173
+ psubw m3, m4174
+ mova [r2], m0175
+ mova [r2 + r3], m1176
+ mova [r2 + r3 * 2], m2177
+ mova [r2 + r4], m3178
+179
+ mova m0, [r0 + mmsize]180
+ mova m1, [r0 + r1 + mmsize]181
+ mova m2, [r0 + r1 * 2 + mmsize]182
+ mova m3, [r0 + r5 + mmsize]183
+ psllw m0, (14 - BIT_DEPTH)184
+ psllw m1, (14 - BIT_DEPTH)185
+ psllw m2, (14 - BIT_DEPTH)186
+ psllw m3, (14 - BIT_DEPTH)187
+ psubw m0, m4188
+ psubw m1, m4189
+ psubw m2, m4190
+ psubw m3, m4191
+ mova [r2 + mmsize], m0192
+ mova [r2 + r3 + mmsize], m1193
+ mova [r2 + r3 * 2 + mmsize], m2194
+ mova [r2 + r4 + mmsize], m3195
+%endmacro196
+197
+%macro P2S_32x4_AVX512 0198
+ movu m0, [r0]199
+ movu m1, [r0 + r1]200
+ movu m2, [r0 + r1 * 2]201
+ movu m3, [r0 + r5]202
+ psllw m0, (14 - BIT_DEPTH)203
+ psllw m1, (14 - BIT_DEPTH)204
+ psllw m2, (14 - BIT_DEPTH)205
+ psllw m3, (14 - BIT_DEPTH)206
+ psubw m0, m4207
+ psubw m1, m4208
+ psubw m2, m4209
+ psubw m3, m4210
+ movu [r2], m0211
+ movu [r2 + r3], m1212
+ movu [r2 + r3 * 2], m2213
+ movu [r2 + r4], m3214
+%endmacro215
+216
+%macro P2S_ALIGNED_32x4_AVX512 0217
+ mova m0, [r0]218
+ mova m1, [r0 + r1]219
+ mova m2, [r0 + r1 * 2]220
+ mova m3, [r0 + r5]221
+ psllw m0, (14 - BIT_DEPTH)222
+ psllw m1, (14 - BIT_DEPTH)223
+ psllw m2, (14 - BIT_DEPTH)224
+ psllw m3, (14 - BIT_DEPTH)225
+ psubw m0, m4226
+ psubw m1, m4227
+ psubw m2, m4228
+ psubw m3, m4229
+ mova [r2], m0230
+ mova [r2 + r3], m1231
+ mova [r2 + r3 * 2], m2232
+ mova [r2 + r4], m3233
+%endmacro234
+235
+%macro P2S_48x4_AVX512 0236
+ movu m0, [r0]237
+ movu m1, [r0 + r1]238
+ movu m2, [r0 + r1 * 2]239
+ movu m3, [r0 + r5]240
+ psllw m0, (14 - BIT_DEPTH)241
+ psllw m1, (14 - BIT_DEPTH)242
+ psllw m2, (14 - BIT_DEPTH)243
+ psllw m3, (14 - BIT_DEPTH)244
+ psubw m0, m4245
+ psubw m1, m4246
+ psubw m2, m4247
+ psubw m3, m4248
+ movu [r2], m0249
+ movu [r2 + r3], m1250
+ movu [r2 + r3 * 2], m2251
+ movu [r2 + r4], m3252
+253
+ movu ym0, [r0 + mmsize]254
+ movu ym1, [r0 + r1 + mmsize]255
+ movu ym2, [r0 + r1 * 2 + mmsize]256
+ movu ym3, [r0 + r5 + mmsize]257
+ psllw ym0, (14 - BIT_DEPTH)258
+ psllw ym1, (14 - BIT_DEPTH)259
+ psllw ym2, (14 - BIT_DEPTH)260
+ psllw ym3, (14 - BIT_DEPTH)261
+ psubw ym0, ym4262
+ psubw ym1, ym4263
+ psubw ym2, ym4264
+ psubw ym3, ym4265
+ movu [r2 + mmsize], ym0266
+ movu [r2 + r3 + mmsize], ym1267
+ movu [r2 + r3 * 2 + mmsize], ym2268
+ movu [r2 + r4 + mmsize], ym3269
+%endmacro270
+271
+%macro P2S_ALIGNED_48x4_AVX512 0272
+ mova m0, [r0]273
+ mova m1, [r0 + r1]274
+ mova m2, [r0 + r1 * 2]275
+ mova m3, [r0 + r5]276
+ psllw m0, (14 - BIT_DEPTH)277
+ psllw m1, (14 - BIT_DEPTH)278
+ psllw m2, (14 - BIT_DEPTH)279
+ psllw m3, (14 - BIT_DEPTH)280
+ psubw m0, m4281
+ psubw m1, m4282
+ psubw m2, m4283
+ psubw m3, m4284
+ mova [r2], m0285
+ mova [r2 + r3], m1286
+ mova [r2 + r3 * 2], m2287
+ mova [r2 + r4], m3288
+289
+ mova ym0, [r0 + mmsize]290
+ mova ym1, [r0 + r1 + mmsize]291
+ mova ym2, [r0 + r1 * 2 + mmsize]292
+ mova ym3, [r0 + r5 + mmsize]293
+ psllw ym0, (14 - BIT_DEPTH)294
+ psllw ym1, (14 - BIT_DEPTH)295
+ psllw ym2, (14 - BIT_DEPTH)296
+ psllw ym3, (14 - BIT_DEPTH)297
+ psubw ym0, ym4298
+ psubw ym1, ym4299
+ psubw ym2, ym4300
+ psubw ym3, ym4301
+ mova [r2 + mmsize], ym0302
+ mova [r2 + r3 + mmsize], ym1303
+ mova [r2 + r3 * 2 + mmsize], ym2304
+ mova [r2 + r4 + mmsize], ym3305
+%endmacro306
+307
+;-----------------------------------------------------------------------------308
+; void filterPixelToShort(pixel *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride)309
+;-----------------------------------------------------------------------------310
+INIT_ZMM avx512311
+cglobal filterPixelToShort_64x16, 4, 6, 5312
+ add r1d, r1d313
+ add r3d, r3d314
+ lea r4, [r3 * 3]315
+ lea r5, [r1 * 3]316
+317
+ ; load constant318
+ vbroadcasti32x8 m4, [pw_2000]319
+%rep 3320
+ P2S_64x4_AVX512321
+ lea r0, [r0 + r1 * 4]322
+ lea r2, [r2 + r3 * 4]323
+%endrep324
+ P2S_64x4_AVX512325
+ RET326
+327
+328
+INIT_ZMM avx512329
+cglobal filterPixelToShort_64x32, 4, 6, 5330
+ add r1d, r1d331
+ add r3d, r3d332
+ lea r4, [r3 * 3]333
+ lea r5, [r1 * 3]334
+335
+ ; load constant336
+ vbroadcasti32x8 m4, [pw_2000]337
+%rep 7338
+ P2S_64x4_AVX512339
+ lea r0, [r0 + r1 * 4]340
+ lea r2, [r2 + r3 * 4]341
+%endrep342
+ P2S_64x4_AVX512343
+ RET344
+345
+INIT_ZMM avx512346
+cglobal filterPixelToShort_64x48, 4, 6, 5347
+ add r1d, r1d348
+ add r3d, r3d349
+ lea r4, [r3 * 3]350
+ lea r5, [r1 * 3]351
+352
+ ; load constant353
+ vbroadcasti32x8 m4, [pw_2000]354
+%rep 11355
+ P2S_64x4_AVX512356
+ lea r0, [r0 + r1 * 4]357
+ lea r2, [r2 + r3 * 4]358
+%endrep359
+ P2S_64x4_AVX512360
+ RET361
+362
+INIT_ZMM avx512363
+cglobal filterPixelToShort_64x64, 4, 6, 5364
+ add r1d, r1d365
+ add r3d, r3d366
+ lea r4, [r3 * 3]367
+ lea r5, [r1 * 3]368
+369
+ ; load constant370
+ vbroadcasti32x8 m4, [pw_2000]371
+%rep 15372
+ P2S_64x4_AVX512373
+ lea r0, [r0 + r1 * 4]374
+ lea r2, [r2 + r3 * 4]375
+%endrep376
+ P2S_64x4_AVX512377
+ RET378
+379
+INIT_ZMM avx512380
+cglobal filterPixelToShort_32x8, 4, 6, 5381
+ add r1d, r1d382
+ add r3d, r3d383
+ lea r4, [r3 * 3]384
+ lea r5, [r1 * 3]385
+386
+ ; load constant387
+ vbroadcasti32x8 m4, [pw_2000]388
+ P2S_32x4_AVX512389
+ lea r0, [r0 + r1 * 4]390
+ lea r2, [r2 + r3 * 4]391
+ P2S_32x4_AVX512392
+ RET393
+394
+INIT_ZMM avx512395
+cglobal filterPixelToShort_32x16, 4, 6, 5396
+ add r1d, r1d397
+ add r3d, r3d398
+ lea r4, [r3 * 3]399
+ lea r5, [r1 * 3]400
+401
+ ; load constant402
+ vbroadcasti32x8 m4, [pw_2000]403
+%rep 3404
+ P2S_32x4_AVX512405
+ lea r0, [r0 + r1 * 4]406
+ lea r2, [r2 + r3 * 4]407
+%endrep408
+ P2S_32x4_AVX512409
+ RET410
+411
+INIT_ZMM avx512412
+cglobal filterPixelToShort_32x24, 4, 6, 5413
+ add r1d, r1d414
+ add r3d, r3d415
+ lea r4, [r3 * 3]416
+ lea r5, [r1 * 3]417
+418
+ ; load constant419
+ vbroadcasti32x8 m4, [pw_2000]420
+%rep 5421
+ P2S_32x4_AVX512422
+ lea r0, [r0 + r1 * 4]423
+ lea r2, [r2 + r3 * 4]424
+%endrep425
+ P2S_32x4_AVX512426
+ RET427
+428
+INIT_ZMM avx512429
+cglobal filterPixelToShort_32x32, 4, 6, 5430
+ add r1d, r1d431
+ add r3d, r3d432
+ lea r4, [r3 * 3]433
+ lea r5, [r1 * 3]434
+435
+ ; load constant436
+ vbroadcasti32x8 m4, [pw_2000]437
+%rep 7438
+ P2S_32x4_AVX512439
+ lea r0, [r0 + r1 * 4]440
+ lea r2, [r2 + r3 * 4]441
+%endrep442
+ P2S_32x4_AVX512443
+ RET444
+445
+INIT_ZMM avx512446
+cglobal filterPixelToShort_32x48, 4, 6, 5447
+ add r1d, r1d448
+ add r3d, r3d449
+ lea r4, [r3 * 3]450
+ lea r5, [r1 * 3]451
+452
+ ; load constant453
+ vbroadcasti32x8 m4, [pw_2000]454
+%rep 11455
+ P2S_32x4_AVX512456
+ lea r0, [r0 + r1 * 4]457
+ lea r2, [r2 + r3 * 4]458
+%endrep459
+ P2S_32x4_AVX512460
+ RET461
+462
+INIT_ZMM avx512463
+cglobal filterPixelToShort_32x64, 4, 6, 5464
+ add r1d, r1d465
+ add r3d, r3d466
+ lea r4, [r3 * 3]467
+ lea r5, [r1 * 3]468
+469
+ ; load constant470
+ vbroadcasti32x8 m4, [pw_2000]471
+%rep 15472
+ P2S_32x4_AVX512473
+ lea r0, [r0 + r1 * 4]474
+ lea r2, [r2 + r3 * 4]475
+%endrep476
+ P2S_32x4_AVX512477
+ RET478
479
+INIT_ZMM avx512480
+cglobal filterPixelToShort_48x64, 4, 6, 5481
+ add r1d, r1d482
+ add r3d, r3d483
+ lea r4, [r3 * 3]484
+ lea r5, [r1 * 3]485
+486
+ ; load constant487
+ vbroadcasti32x8 m4, [pw_2000]488
+%rep 15489
+ P2S_48x4_AVX512490
+ lea r0, [r0 + r1 * 4]491
+ lea r2, [r2 + r3 * 4]492
+%endrep493
+ P2S_48x4_AVX512494
+ RET495
+496
+INIT_ZMM avx512497
+cglobal filterPixelToShort_aligned_64x16, 4, 6, 5498
+ add r1d, r1d499
+ add r3d, r3d500
+ lea r4, [r3 * 3]501
+ lea r5, [r1 * 3]502
+503
+ ; load constant504
+ vbroadcasti32x8 m4, [pw_2000]505
+%rep 3506
+ P2S_ALIGNED_64x4_AVX512507
+ lea r0, [r0 + r1 * 4]508
+ lea r2, [r2 + r3 * 4]509
+%endrep510
+ P2S_ALIGNED_64x4_AVX512511
+ RET512
+513
+514
+INIT_ZMM avx512515
+cglobal filterPixelToShort_aligned_64x32, 4, 6, 5516
+ add r1d, r1d517
+ add r3d, r3d518
+ lea r4, [r3 * 3]519
+ lea r5, [r1 * 3]520
+521
+ ; load constant522
+ vbroadcasti32x8 m4, [pw_2000]523
+%rep 7524
+ P2S_ALIGNED_64x4_AVX512525
+ lea r0, [r0 + r1 * 4]526
+ lea r2, [r2 + r3 * 4]527
+%endrep528
+ P2S_ALIGNED_64x4_AVX512529
+ RET530
+531
+INIT_ZMM avx512532
+cglobal filterPixelToShort_aligned_64x48, 4, 6, 5533
+ add r1d, r1d534
+ add r3d, r3d535
+ lea r4, [r3 * 3]536
+ lea r5, [r1 * 3]537
+538
+ ; load constant539
+ vbroadcasti32x8 m4, [pw_2000]540
+%rep 11541
+ P2S_ALIGNED_64x4_AVX512542
+ lea r0, [r0 + r1 * 4]543
+ lea r2, [r2 + r3 * 4]544
+%endrep545
+ P2S_ALIGNED_64x4_AVX512546
+ RET547
+548
+INIT_ZMM avx512549
+cglobal filterPixelToShort_aligned_64x64, 4, 6, 5550
+ add r1d, r1d551
+ add r3d, r3d552
+ lea r4, [r3 * 3]553
+ lea r5, [r1 * 3]554
+555
+ ; load constant556
+ vbroadcasti32x8 m4, [pw_2000]557
+%rep 15558
+ P2S_ALIGNED_64x4_AVX512559
+ lea r0, [r0 + r1 * 4]560
+ lea r2, [r2 + r3 * 4]561
+%endrep562
+ P2S_ALIGNED_64x4_AVX512563
+ RET564
+565
+INIT_ZMM avx512566
+cglobal filterPixelToShort_aligned_32x8, 4, 6, 5567
+ add r1d, r1d568
+ add r3d, r3d569
+ lea r4, [r3 * 3]570
+ lea r5, [r1 * 3]571
+572
+ ; load constant573
+ vbroadcasti32x8 m4, [pw_2000]574
+ P2S_ALIGNED_32x4_AVX512575
+ lea r0, [r0 + r1 * 4]576
+ lea r2, [r2 + r3 * 4]577
+ P2S_ALIGNED_32x4_AVX512578
+ RET579
+580
+INIT_ZMM avx512581
+cglobal filterPixelToShort_aligned_32x16, 4, 6, 5582
+ add r1d, r1d583
+ add r3d, r3d584
+ lea r4, [r3 * 3]585
+ lea r5, [r1 * 3]586
+587
+ ; load constant588
+ vbroadcasti32x8 m4, [pw_2000]589
+%rep 3590
+ P2S_ALIGNED_32x4_AVX512591
+ lea r0, [r0 + r1 * 4]592
+ lea r2, [r2 + r3 * 4]593
+%endrep594
+ P2S_ALIGNED_32x4_AVX512595
+ RET596
+597
+INIT_ZMM avx512598
+cglobal filterPixelToShort_aligned_32x24, 4, 6, 5599
+ add r1d, r1d600
+ add r3d, r3d601
+ lea r4, [r3 * 3]602
+ lea r5, [r1 * 3]603
+604
+ ; load constant605
+ vbroadcasti32x8 m4, [pw_2000]606
+%rep 5607
+ P2S_ALIGNED_32x4_AVX512608
+ lea r0, [r0 + r1 * 4]609
+ lea r2, [r2 + r3 * 4]610
+%endrep611
+ P2S_ALIGNED_32x4_AVX512612
+ RET613
+614
+INIT_ZMM avx512615
+cglobal filterPixelToShort_aligned_32x32, 4, 6, 5616
+ add r1d, r1d617
+ add r3d, r3d618
+ lea r4, [r3 * 3]619
+ lea r5, [r1 * 3]620
+621
+ ; load constant622
+ vbroadcasti32x8 m4, [pw_2000]623
+%rep 7624
+ P2S_ALIGNED_32x4_AVX512625
+ lea r0, [r0 + r1 * 4]626
+ lea r2, [r2 + r3 * 4]627
+%endrep628
+ P2S_ALIGNED_32x4_AVX512629
+ RET630
+631
+INIT_ZMM avx512632
+cglobal filterPixelToShort_aligned_32x48, 4, 6, 5633
+ add r1d, r1d634
+ add r3d, r3d635
+ lea r4, [r3 * 3]636
+ lea r5, [r1 * 3]637
+638
+ ; load constant639
+ vbroadcasti32x8 m4, [pw_2000]640
+%rep 11641
+ P2S_ALIGNED_32x4_AVX512642
+ lea r0, [r0 + r1 * 4]643
+ lea r2, [r2 + r3 * 4]644
+%endrep645
+ P2S_ALIGNED_32x4_AVX512646
+ RET647
+648
+INIT_ZMM avx512649
+cglobal filterPixelToShort_aligned_32x64, 4, 6, 5650
+ add r1d, r1d651
+ add r3d, r3d652
+ lea r4, [r3 * 3]653
+ lea r5, [r1 * 3]654
+655
+ ; load constant656
+ vbroadcasti32x8 m4, [pw_2000]657
+%rep 15658
+ P2S_ALIGNED_32x4_AVX512659
+ lea r0, [r0 + r1 * 4]660
+ lea r2, [r2 + r3 * 4]661
+%endrep662
+ P2S_ALIGNED_32x4_AVX512663
+ RET664
+665
+INIT_ZMM avx512666
+cglobal filterPixelToShort_aligned_48x64, 4, 6, 5667
+ add r1d, r1d668
+ add r3d, r3d669
+ lea r4, [r3 * 3]670
+ lea r5, [r1 * 3]671
+672
+ ; load constant673
+ vbroadcasti32x8 m4, [pw_2000]674
+%rep 15675
+ P2S_ALIGNED_48x4_AVX512676
+ lea r0, [r0 + r1 * 4]677
+ lea r2, [r2 + r3 * 4]678
+%endrep679
+ P2S_ALIGNED_48x4_AVX512680
+ RET681
+;-----------------------------------------------------------------------------------------------------------------------------682
+;p2s and p2s_aligned avx512 code end683
+;-----------------------------------------------------------------------------------------------------------------------------684
685
%macro PROCESS_LUMA_VER_W4_4R 0686
movq m0, [r0]687
688
jnz .loop689
RET690
691
+;-------------------------------------------------------------------------------------------------------------692
+;ipfilter_chroma_avx512 code start693
+;-------------------------------------------------------------------------------------------------------------694
+;-------------------------------------------------------------------------------------------------------------695
+; avx512 chroma_hpp code start696
+;-------------------------------------------------------------------------------------------------------------697
+%macro PROCESS_IPFILTER_CHROMA_PP_8x4_AVX512 0698
+ ; register map699
+ ; m0 , m1 interpolate coeff700
+ ; m2 , m3 shuffle order table701
+ ; m4 - pd_32702
+ ; m5 - zero703
+ ; m6 - pw_pixel_max704
+705
+ movu xm7, [r0]706
+ vinserti32x4 m7, [r0 + r1], 1707
+ vinserti32x4 m7, [r0 + 2 * r1], 2708
+ vinserti32x4 m7, [r0 + r6], 3709
+710
+ pshufb m9, m7, m3711
+ pshufb m7, m2712
+ pmaddwd m7, m0713
+ pmaddwd m9, m1714
+ paddd m7, m9715
+ paddd m7, m4716
+ psrad m7, 6717
+718
+ movu xm8, [r0 + 8]719
+ vinserti32x4 m8, [r0 + r1 + 8], 1720
+ vinserti32x4 m8, [r0 + 2 * r1 + 8], 2721
+ vinserti32x4 m8, [r0 + r6 + 8], 3722
+723
+ pshufb m9, m8, m3724
+ pshufb m8, m2725
+ pmaddwd m8, m0726
+ pmaddwd m9, m1727
+ paddd m8, m9728
+ paddd m8, m4729
+ psrad m8, 6730
+731
+ packusdw m7, m8732
+ CLIPW m7, m5, m6733
+ pshufb m7, m10734
+ movu [r2], xm7735
+ vextracti32x4 [r2 + r3], m7, 1736
+ vextracti32x4 [r2 + 2 * r3], m7, 2737
+ vextracti32x4 [r2 + r7], m7, 3738
+%endmacro739
+740
+%macro PROCESS_IPFILTER_CHROMA_PP_16x2_AVX512 0741
+ ; register map742
+ ; m0 , m1 interpolate coeff743
+ ; m2 , m3 shuffle order table744
+ ; m4 - pd_32745
+ ; m5 - zero746
+ ; m6 - pw_pixel_max747
+748
+ movu ym7, [r0]749
+ vinserti32x8 m7, [r0 + r1], 1750
+ movu ym8, [r0 + 8]751
+ vinserti32x8 m8, [r0 + r1 + 8], 1752
+753
+ pshufb m9, m7, m3754
+ pshufb m7, m2755
+ pmaddwd m7, m0756
+ pmaddwd m9, m1757
+ paddd m7, m9758
+ paddd m7, m4759
+ psrad m7, 6760
+761
+ pshufb m9, m8, m3762
+ pshufb m8, m2763
+ pmaddwd m8, m0764
+ pmaddwd m9, m1765
+ paddd m8, m9766
+ paddd m8, m4767
+ psrad m8, 6768
+769
+ packusdw m7, m8770
+ CLIPW m7, m5, m6771
+ pshufb m7, m10772
+ movu [r2], ym7773
+ vextracti32x8 [r2 + r3], m7, 1774
+%endmacro775
+776
+%macro PROCESS_IPFILTER_CHROMA_PP_24x4_AVX512 0777
+ ; register map778
+ ; m0 , m1 interpolate coeff779
+ ; m2 , m3 shuffle order table780
+ ; m4 - pd_32781
+ ; m5 - zero782
+ ; m6 - pw_pixel_max783
+784
+ movu ym7, [r0]785
+ vinserti32x8 m7, [r0 + r1], 1786
+ movu ym8, [r0 + 8]787
+ vinserti32x8 m8, [r0 + r1 + 8], 1788
+789
+ pshufb m9, m7, m3790
+ pshufb m7, m2791
+ pmaddwd m7, m0792
+ pmaddwd m9, m1793
+ paddd m7, m9794
+ paddd m7, m4795
+ psrad m7, 6796
+797
+ pshufb m9, m8, m3798
+ pshufb m8, m2799
+ pmaddwd m8, m0800
+ pmaddwd m9, m1801
+ paddd m8, m9802
+ paddd m8, m4803
+ psrad m8, 6804
+805
+ packusdw m7, m8806
+ CLIPW m7, m5, m6807
+ pshufb m7, m10808
+ movu [r2], ym7809
+ vextracti32x8 [r2 + r3], m7, 1810
+811
+ movu ym7, [r0 + 2 * r1]812
+ vinserti32x8 m7, [r0 + r6], 1813
+ movu ym8, [r0 + 2 * r1 + 8]814
+ vinserti32x8 m8, [r0 + r6 + 8], 1815
+816
+ pshufb m9, m7, m3817
+ pshufb m7, m2818
+ pmaddwd m7, m0819
+ pmaddwd m9, m1820
+ paddd m7, m9821
+ paddd m7, m4822
+ psrad m7, 6823
+824
+ pshufb m9, m8, m3825
+ pshufb m8, m2826
+ pmaddwd m8, m0827
+ pmaddwd m9, m1828
+ paddd m8, m9829
+ paddd m8, m4830
+ psrad m8, 6831
+832
+ packusdw m7, m8833
+ CLIPW m7, m5, m6834
+ pshufb m7, m10835
+ movu [r2 + 2 * r3], ym7836
+ vextracti32x8 [r2 + r7], m7, 1837
+838
+ movu xm7, [r0 + mmsize/2]839
+ vinserti32x4 m7, [r0 + r1 + mmsize/2], 1840
+ vinserti32x4 m7, [r0 + 2 * r1 + mmsize/2], 2841
+ vinserti32x4 m7, [r0 + r6 + mmsize/2], 3842
+843
+ pshufb m9, m7, m3844
+ pshufb m7, m2845
+ pmaddwd m7, m0846
+ pmaddwd m9, m1847
+ paddd m7, m9848
+ paddd m7, m4849
+ psrad m7, 6850
+851
+ movu xm8, [r0 + mmsize/2 + 8]852
+ vinserti32x4 m8, [r0 + r1 + mmsize/2 + 8], 1853
+ vinserti32x4 m8, [r0 + 2 * r1 + mmsize/2 + 8], 2854
+ vinserti32x4 m8, [r0 + r6 + mmsize/2 + 8], 3855
+856
+ pshufb m9, m8, m3857
+ pshufb m8, m2858
+ pmaddwd m8, m0859
+ pmaddwd m9, m1860
+ paddd m8, m9861
+ paddd m8, m4862
+ psrad m8, 6863
+864
+ packusdw m7, m8865
+ CLIPW m7, m5, m6866
+ pshufb m7, m10867
+ movu [r2 + mmsize/2], xm7868
+ vextracti32x4 [r2 + r3 + mmsize/2], m7, 1869
+ vextracti32x4 [r2 + 2 * r3 + mmsize/2], m7, 2870
+ vextracti32x4 [r2 + r7 + mmsize/2], m7, 3871
+%endmacro872
+873
+%macro PROCESS_IPFILTER_CHROMA_PP_32x2_AVX512 0874
+ ; register map875
+ ; m0 , m1 interpolate coeff876
+ ; m2 , m3 shuffle order table877
+ ; m4 - pd_32878
+ ; m5 - zero879
+ ; m6 - pw_pixel_max880
+881
+ movu m7, [r0]882
+ movu m8, [r0 + 8]883
+884
+ pshufb m9, m7, m3885
+ pshufb m7, m2886
+ pmaddwd m7, m0887
+ pmaddwd m9, m1888
+ paddd m7, m9889
+ paddd m7, m4890
+ psrad m7, 6891
+892
+ pshufb m9, m8, m3893
+ pshufb m8, m2894
+ pmaddwd m8, m0895
+ pmaddwd m9, m1896
+ paddd m8, m9897
+ paddd m8, m4898
+ psrad m8, 6899
+900
+ packusdw m7, m8901
+ CLIPW m7, m5, m6902
+ pshufb m7, m10903
+ movu [r2], m7904
+905
+ movu m7, [r0 + r1]906
+ movu m8, [r0 + r1 + 8]907
+908
+ pshufb m9, m7, m3909
+ pshufb m7, m2910
+ pmaddwd m7, m0911
+ pmaddwd m9, m1912
+ paddd m7, m9913
+ paddd m7, m4914
+ psrad m7, 6915
+916
+ pshufb m9, m8, m3917
+ pshufb m8, m2918
+ pmaddwd m8, m0919
+ pmaddwd m9, m1920
+ paddd m8, m9921
+ paddd m8, m4922
+ psrad m8, 6923
+924
+ packusdw m7, m8925
+ CLIPW m7, m5, m6926
+ pshufb m7, m10927
+ movu [r2 + r3], m7928
+%endmacro929
+930
+%macro PROCESS_IPFILTER_CHROMA_PP_48x2_AVX512 0931
+ ; register map932
+ ; m0 , m1 interpolate coeff933
+ ; m2 , m3 shuffle order table934
+ ; m4 - pd_32935
+ ; m5 - zero936
+ ; m6 - pw_pixel_max937
+938
+ movu m7, [r0]939
+ movu m8, [r0 + 8]940
+941
+ pshufb m9, m7, m3942
+ pshufb m7, m2943
+ pmaddwd m7, m0944
+ pmaddwd m9, m1945
+ paddd m7, m9946
+ paddd m7, m4947
+ psrad m7, 6948
+949
+ pshufb m9, m8, m3950
+ pshufb m8, m2951
+ pmaddwd m8, m0952
+ pmaddwd m9, m1953
+ paddd m8, m9954
+ paddd m8, m4955
+ psrad m8, 6956
+957
+ packusdw m7, m8958
+ CLIPW m7, m5, m6959
+ pshufb m7, m10960
+ movu [r2], m7961
+962
+ movu m7, [r0 + r1]963
+ movu m8, [r0 + r1 + 8]964
+965
+ pshufb m9, m7, m3966
+ pshufb m7, m2967
+ pmaddwd m7, m0968
+ pmaddwd m9, m1969
+ paddd m7, m9970
+ paddd m7, m4971
+ psrad m7, 6972
+973
+ pshufb m9, m8, m3974
+ pshufb m8, m2975
+ pmaddwd m8, m0976
+ pmaddwd m9, m1977
+ paddd m8, m9978
+ paddd m8, m4979
+ psrad m8, 6980
+981
+ packusdw m7, m8982
+ CLIPW m7, m5, m6983
+ pshufb m7, m10984
+ movu [r2 + r3], m7985
+986
+ movu ym7, [r0 + mmsize]987
+ vinserti32x8 m7, [r0 + r1 + mmsize], 1988
+ movu ym8, [r0 + mmsize + 8]989
+ vinserti32x8 m8, [r0 + r1 + mmsize + 8], 1990
+991
+ pshufb m9, m7, m3992
+ pshufb m7, m2993
+ pmaddwd m7, m0994
+ pmaddwd m9, m1995
+ paddd m7, m9996
+ paddd m7, m4997
+ psrad m7, 6998
+999
+ pshufb m9, m8, m31000
+ pshufb m8, m21001
+ pmaddwd m8, m01002
+ pmaddwd m9, m11003
+ paddd m8, m91004
+ paddd m8, m41005
+ psrad m8, 61006
+1007
+ packusdw m7, m81008
+ CLIPW m7, m5, m61009
+ pshufb m7, m101010
+ movu [r2 + mmsize], ym71011
+ vextracti32x8 [r2 + r3 + mmsize], m7, 11012
+%endmacro1013
+1014
+%macro PROCESS_IPFILTER_CHROMA_PP_64x2_AVX512 01015
+ ; register map1016
+ ; m0 , m1 interpolate coeff1017
+ ; m2 , m3 shuffle order table1018
+ ; m4 - pd_321019
+ ; m5 - zero1020
+ ; m6 - pw_pixel_max1021
+1022
+ movu m7, [r0]1023
+ movu m8, [r0 + 8]1024
+1025
+ pshufb m9, m7, m31026
+ pshufb m7, m21027
+ pmaddwd m7, m01028
+ pmaddwd m9, m11029
+ paddd m7, m91030
+ paddd m7, m41031
+ psrad m7, 61032
+1033
+ pshufb m9, m8, m31034
+ pshufb m8, m21035
+ pmaddwd m8, m01036
+ pmaddwd m9, m11037
+ paddd m8, m91038
+ paddd m8, m41039
+ psrad m8, 61040
+1041
+ packusdw m7, m81042
+ CLIPW m7, m5, m61043
+ pshufb m7, m101044
+ movu [r2], m71045
+1046
+ movu m7, [r0 + mmsize]1047
+ movu m8, [r0 + mmsize + 8]1048
+1049
+ pshufb m9, m7, m31050
+ pshufb m7, m21051
+ pmaddwd m7, m01052
+ pmaddwd m9, m11053
+ paddd m7, m91054
+ paddd m7, m41055
+ psrad m7, 61056
+1057
+ pshufb m9, m8, m31058
+ pshufb m8, m21059
+ pmaddwd m8, m01060
+ pmaddwd m9, m11061
+ paddd m8, m91062
+ paddd m8, m41063
+ psrad m8, 61064
+1065
+ packusdw m7, m81066
+ CLIPW m7, m5, m61067
+ pshufb m7, m101068
+ movu [r2 + mmsize], m71069
+1070
+ movu m7, [r0 + r1]1071
+ movu m8, [r0 + r1 + 8]1072
+1073
+ pshufb m9, m7, m31074
+ pshufb m7, m21075
+ pmaddwd m7, m01076
+ pmaddwd m9, m11077
+ paddd m7, m91078
+ paddd m7, m41079
+ psrad m7, 61080
+1081
+ pshufb m9, m8, m31082
+ pshufb m8, m21083
+ pmaddwd m8, m01084
+ pmaddwd m9, m11085
+ paddd m8, m91086
+ paddd m8, m41087
+ psrad m8, 61088
+1089
+ packusdw m7, m81090
+ CLIPW m7, m5, m61091
+ pshufb m7, m101092
+ movu [r2 + r3], m71093
+1094
+ movu m7, [r0 + r1 + mmsize]1095
+ movu m8, [r0 + r1 + mmsize + 8]1096
+1097
+ pshufb m9, m7, m31098
+ pshufb m7, m21099
+ pmaddwd m7, m01100
+ pmaddwd m9, m11101
+ paddd m7, m91102
+ paddd m7, m41103
+ psrad m7, 61104
+1105
+ pshufb m9, m8, m31106
+ pshufb m8, m21107
+ pmaddwd m8, m01108
+ pmaddwd m9, m11109
+ paddd m8, m91110
+ paddd m8, m41111
+ psrad m8, 61112
+1113
+ packusdw m7, m81114
+ CLIPW m7, m5, m61115
+ pshufb m7, m101116
+ movu [r2 + r3 + mmsize], m71117
+%endmacro1118
+;-------------------------------------------------------------------------------------------------------------1119
+; void interp_4tap_horiz_pp(pixel *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx1120
+;-------------------------------------------------------------------------------------------------------------1121
+%if ARCH_X86_641122
+INIT_ZMM avx5121123
+cglobal interp_4tap_horiz_pp_8x4, 5,8,111124
+ add r1d, r1d1125
+ add r3d, r3d1126
+ sub r0, 21127
+ mov r4d, r4m1128
+ lea r6, [3 * r1]1129
+ lea r7, [3 * r3]1130
+%ifdef PIC1131
+ lea r5, [tab_ChromaCoeff]1132
+ vpbroadcastd m0, [r5 + r4 * 8]1133
+ vpbroadcastd m1, [r5 + r4 * 8 + 4]1134
+%else1135
+ vpbroadcastd m0, [tab_ChromaCoeff + r4 * 8]1136
+ vpbroadcastd m1, [tab_ChromaCoeff + r4 * 8 + 4]1137
+%endif1138
+ vbroadcasti32x8 m2, [interp8_hpp_shuf1_load_avx512]1139
+ vbroadcasti32x8 m3, [interp8_hpp_shuf2_load_avx512]1140
+ vbroadcasti32x8 m4, [pd_32]1141
+ pxor m5, m51142
+ vbroadcasti32x8 m6, [pw_pixel_max]1143
+ vbroadcasti32x8 m10, [interp8_hpp_shuf1_store_avx512]1144
+1145
+ PROCESS_IPFILTER_CHROMA_PP_8x4_AVX5121146
+ RET1147
+%endif1148
+1149
+%macro IPFILTER_CHROMA_AVX512_8xN 11150
+INIT_ZMM avx5121151
+cglobal interp_4tap_horiz_pp_8x%1, 5,8,111152
+ add r1d, r1d1153
+ add r3d, r3d1154
+ sub r0, 21155
+ mov r4d, r4m1156
+ lea r6, [3 * r1]1157
+ lea r7, [3 * r3]1158
+%ifdef PIC1159
+ lea r5, [tab_ChromaCoeff]1160
+ vpbroadcastd m0, [r5 + r4 * 8]1161
+ vpbroadcastd m1, [r5 + r4 * 8 + 4]1162
+%else1163
+ vpbroadcastd m0, [tab_ChromaCoeff + r4 * 8]1164
+ vpbroadcastd m1, [tab_ChromaCoeff + r4 * 8 + 4]1165
+%endif1166
+ vbroadcasti32x8 m2, [interp8_hpp_shuf1_load_avx512]1167
+ vbroadcasti32x8 m3, [interp8_hpp_shuf2_load_avx512]1168
+ vbroadcasti32x8 m4, [pd_32]1169
+ pxor m5, m51170
+ vbroadcasti32x8 m6, [pw_pixel_max]1171
+ vbroadcasti32x8 m10, [interp8_hpp_shuf1_store_avx512]1172
+1173
+%rep %1/4 - 11174
+ PROCESS_IPFILTER_CHROMA_PP_8x4_AVX5121175
+ lea r0, [r0 + 4 * r1]1176
+ lea r2, [r2 + 4 * r3]1177
+%endrep1178
+ PROCESS_IPFILTER_CHROMA_PP_8x4_AVX5121179
+ RET1180
+%endmacro1181
+1182
+%if ARCH_X86_641183
+IPFILTER_CHROMA_AVX512_8xN 81184
+IPFILTER_CHROMA_AVX512_8xN 121185
+IPFILTER_CHROMA_AVX512_8xN 161186
+IPFILTER_CHROMA_AVX512_8xN 321187
+IPFILTER_CHROMA_AVX512_8xN 641188
+%endif1189
+1190
+%macro IPFILTER_CHROMA_AVX512_16xN 11191
+INIT_ZMM avx5121192
+cglobal interp_4tap_horiz_pp_16x%1, 5,6,111193
+ add r1d, r1d1194
+ add r3d, r3d1195
+ sub r0, 21196
+ mov r4d, r4m1197
+%ifdef PIC1198
+ lea r5, [tab_ChromaCoeff]1199
+ vpbroadcastd m0, [r5 + r4 * 8]1200
+ vpbroadcastd m1, [r5 + r4 * 8 + 4]1201
+%else1202
+ vpbroadcastd m0, [tab_ChromaCoeff + r4 * 8]1203
+ vpbroadcastd m1, [tab_ChromaCoeff + r4 * 8 + 4]1204
+%endif1205
+ vbroadcasti32x8 m2, [interp8_hpp_shuf1_load_avx512]1206
+ vbroadcasti32x8 m3, [interp8_hpp_shuf2_load_avx512]1207
+ vbroadcasti32x8 m4, [pd_32]1208
+ pxor m5, m51209
+ vbroadcasti32x8 m6, [pw_pixel_max]1210
+ vbroadcasti32x8 m10, [interp8_hpp_shuf1_store_avx512]1211
+1212
+%rep %1/2 - 11213
+ PROCESS_IPFILTER_CHROMA_PP_16x2_AVX5121214
+ lea r0, [r0 + 2 * r1]1215
+ lea r2, [r2 + 2 * r3]1216
+%endrep1217
+ PROCESS_IPFILTER_CHROMA_PP_16x2_AVX5121218
+ RET1219
+%endmacro1220
+1221
+%if ARCH_X86_641222
+IPFILTER_CHROMA_AVX512_16xN 41223
+IPFILTER_CHROMA_AVX512_16xN 81224
+IPFILTER_CHROMA_AVX512_16xN 121225
+IPFILTER_CHROMA_AVX512_16xN 161226
+IPFILTER_CHROMA_AVX512_16xN 241227
+IPFILTER_CHROMA_AVX512_16xN 321228
+IPFILTER_CHROMA_AVX512_16xN 641229
+%endif1230
+1231
+%macro IPFILTER_CHROMA_AVX512_24xN 11232
+INIT_ZMM avx5121233
+cglobal interp_4tap_horiz_pp_24x%1, 5,8,111234
+ add r1d, r1d1235
+ add r3d, r3d1236
+ sub r0, 21237
+ mov r4d, r4m1238
+ lea r6, [3 * r1]1239
+ lea r7, [3 * r3]1240
+%ifdef PIC1241
+ lea r5, [tab_ChromaCoeff]1242
+ vpbroadcastd m0, [r5 + r4 * 8]1243
+ vpbroadcastd m1, [r5 + r4 * 8 + 4]1244
+%else1245
+ vpbroadcastd m0, [tab_ChromaCoeff + r4 * 8]1246
+ vpbroadcastd m1, [tab_ChromaCoeff + r4 * 8 + 4]1247
+%endif1248
+ vbroadcasti32x8 m2, [interp8_hpp_shuf1_load_avx512]1249
+ vbroadcasti32x8 m3, [interp8_hpp_shuf2_load_avx512]1250
+ vbroadcasti32x8 m4, [pd_32]1251
+ pxor m5, m51252
+ vbroadcasti32x8 m6, [pw_pixel_max]1253
+ vbroadcasti32x8 m10, [interp8_hpp_shuf1_store_avx512]1254
+1255
+%rep %1/4 - 11256
+ PROCESS_IPFILTER_CHROMA_PP_24x4_AVX5121257
+ lea r0, [r0 + 4 * r1]1258
+ lea r2, [r2 + 4 * r3]1259
+%endrep1260
+ PROCESS_IPFILTER_CHROMA_PP_24x4_AVX5121261
+ RET1262
+%endmacro1263
+1264
+%if ARCH_X86_641265
+IPFILTER_CHROMA_AVX512_24xN 321266
+IPFILTER_CHROMA_AVX512_24xN 641267
+%endif1268
+1269
+%macro IPFILTER_CHROMA_AVX512_32xN 11270
+INIT_ZMM avx5121271
+cglobal interp_4tap_horiz_pp_32x%1, 5,6,111272
+ add r1d, r1d1273
+ add r3d, r3d1274
+ sub r0, 21275
+ mov r4d, r4m1276
+%ifdef PIC1277
+ lea r5, [tab_ChromaCoeff]1278
+ vpbroadcastd m0, [r5 + r4 * 8]1279
+ vpbroadcastd m1, [r5 + r4 * 8 + 4]1280
+%else1281
+ vpbroadcastd m0, [tab_ChromaCoeff + r4 * 8]1282
+ vpbroadcastd m1, [tab_ChromaCoeff + r4 * 8 + 4]1283
+%endif1284
+ vbroadcasti32x8 m2, [interp8_hpp_shuf1_load_avx512]1285
+ vbroadcasti32x8 m3, [interp8_hpp_shuf2_load_avx512]1286
+ vbroadcasti32x8 m4, [pd_32]1287
+ pxor m5, m51288
+ vbroadcasti32x8 m6, [pw_pixel_max]1289
+ vbroadcasti32x8 m10, [interp8_hpp_shuf1_store_avx512]1290
+1291
+%rep %1/2 - 11292
+ PROCESS_IPFILTER_CHROMA_PP_32x2_AVX5121293
+ lea r0, [r0 + 2 * r1]1294
+ lea r2, [r2 + 2 * r3]1295
+%endrep1296
+ PROCESS_IPFILTER_CHROMA_PP_32x2_AVX5121297
+ RET1298
+%endmacro1299
+1300
+%if ARCH_X86_641301
+IPFILTER_CHROMA_AVX512_32xN 81302
+IPFILTER_CHROMA_AVX512_32xN 161303
+IPFILTER_CHROMA_AVX512_32xN 241304
+IPFILTER_CHROMA_AVX512_32xN 321305
+IPFILTER_CHROMA_AVX512_32xN 481306
+IPFILTER_CHROMA_AVX512_32xN 641307
+%endif1308
+1309
+%macro IPFILTER_CHROMA_AVX512_64xN 11310
+INIT_ZMM avx5121311
+cglobal interp_4tap_horiz_pp_64x%1, 5,6,111312
+ add r1d, r1d1313
+ add r3d, r3d1314
+ sub r0, 21315
+ mov r4d, r4m1316
+%ifdef PIC1317
+ lea r5, [tab_ChromaCoeff]1318
+ vpbroadcastd m0, [r5 + r4 * 8]1319
+ vpbroadcastd m1, [r5 + r4 * 8 + 4]1320
+%else1321
+ vpbroadcastd m0, [tab_ChromaCoeff + r4 * 8]1322
+ vpbroadcastd m1, [tab_ChromaCoeff + r4 * 8 + 4]1323
+%endif1324
+ vbroadcasti32x8 m2, [interp8_hpp_shuf1_load_avx512]1325
+ vbroadcasti32x8 m3, [interp8_hpp_shuf2_load_avx512]1326
+ vbroadcasti32x8 m4, [pd_32]1327
+ pxor m5, m51328
+ vbroadcasti32x8 m6, [pw_pixel_max]1329
+ vbroadcasti32x8 m10, [interp8_hpp_shuf1_store_avx512]1330
+1331
+%rep %1/2 - 11332
+ PROCESS_IPFILTER_CHROMA_PP_64x2_AVX5121333
+ lea r0, [r0 + 2 * r1]1334
+ lea r2, [r2 + 2 * r3]1335
+%endrep1336
+ PROCESS_IPFILTER_CHROMA_PP_64x2_AVX5121337
+ RET1338
+%endmacro1339
+1340
+%if ARCH_X86_641341
+IPFILTER_CHROMA_AVX512_64xN 161342
+IPFILTER_CHROMA_AVX512_64xN 321343
+IPFILTER_CHROMA_AVX512_64xN 481344
+IPFILTER_CHROMA_AVX512_64xN 641345
+%endif1346
+1347
+%if ARCH_X86_641348
+INIT_ZMM avx5121349
+cglobal interp_4tap_horiz_pp_48x64, 5,6,111350
+ add r1d, r1d1351
+ add r3d, r3d1352
+ sub r0, 21353
+ mov r4d, r4m1354
+%ifdef PIC1355
+ lea r5, [tab_ChromaCoeff]1356
+ vpbroadcastd m0, [r5 + r4 * 8]1357
+ vpbroadcastd m1, [r5 + r4 * 8 + 4]1358
+%else1359
+ vpbroadcastd m0, [tab_ChromaCoeff + r4 * 8]1360
+ vpbroadcastd m1, [tab_ChromaCoeff + r4 * 8 + 4]1361
+%endif1362
+ vbroadcasti32x8 m2, [interp8_hpp_shuf1_load_avx512]1363
+ vbroadcasti32x8 m3, [interp8_hpp_shuf2_load_avx512]1364
+ vbroadcasti32x8 m4, [pd_32]1365
+ pxor m5, m51366
+ vbroadcasti32x8 m6, [pw_pixel_max]1367
+ vbroadcasti32x8 m10, [interp8_hpp_shuf1_store_avx512]1368
+1369
+%rep 311370
+ PROCESS_IPFILTER_CHROMA_PP_48x2_AVX5121371
+ lea r0, [r0 + 2 * r1]1372
+ lea r2, [r2 + 2 * r3]1373
+%endrep1374
+ PROCESS_IPFILTER_CHROMA_PP_48x2_AVX5121375
+ RET1376
+%endif1377
+;-------------------------------------------------------------------------------------------------------------1378
+; avx512 chroma_hpp code end1379
+;-------------------------------------------------------------------------------------------------------------1380
+;-------------------------------------------------------------------------------------------------------------1381
+; avx512 chroma_vpp code start1382
+;-------------------------------------------------------------------------------------------------------------1383
+%macro PROCESS_CHROMA_VERT_PP_8x8_AVX512 01384
+ movu xm1, [r0]1385
+ lea r6, [r0 + 2 * r1]1386
+ lea r8, [r0 + 4 * r1]1387
+ lea r9, [r8 + 2 * r1]1388
+ vinserti32x4 m1, [r6], 11389
+ vinserti32x4 m1, [r8], 21390
+ vinserti32x4 m1, [r9], 31391
+ movu xm3, [r0 + r1]1392
+ vinserti32x4 m3, [r6 + r1], 11393
+ vinserti32x4 m3, [r8 + r1], 21394
+ vinserti32x4 m3, [r9 + r1], 31395
+ punpcklwd m0, m1, m31396
+ pmaddwd m0, [r5]1397
+ punpckhwd m1, m31398
+ pmaddwd m1, [r5]1399
+1400
+ movu xm4, [r0 + 2 * r1]1401
+ vinserti32x4 m4, [r6 + 2 * r1], 11402
+ vinserti32x4 m4, [r8 + 2 * r1], 21403
+ vinserti32x4 m4, [r9 + 2 * r1], 31404
+ punpcklwd m2, m3, m41405
+ pmaddwd m2, [r5]1406
+ punpckhwd m3, m41407
+ pmaddwd m3, [r5]1408
+1409
+ movu xm5, [r0 + r10]1410
+ vinserti32x4 m5, [r6 + r10], 11411
+ vinserti32x4 m5, [r8 + r10], 21412
+ vinserti32x4 m5, [r9 + r10], 31413
+ punpcklwd m6, m4, m51414
+ pmaddwd m6, [r5 + mmsize]1415
+ paddd m0, m61416
+ punpckhwd m4, m51417
+ pmaddwd m4, [r5 + mmsize]1418
+ paddd m1, m41419
+1420
+ movu xm4, [r0 + 4 * r1]1421
+ vinserti32x4 m4, [r6 + 4 * r1], 11422
+ vinserti32x4 m4, [r8 + 4 * r1], 21423
+ vinserti32x4 m4, [r9 + 4 * r1], 31424
+ punpcklwd m6, m5, m41425
+ pmaddwd m6, [r5 + mmsize]1426
+ paddd m2, m61427
+ punpckhwd m5, m41428
+ pmaddwd m5, [r5 + mmsize]1429
+ paddd m3, m51430
+1431
+ paddd m0, m71432
+ paddd m1, m71433
+ paddd m2, m71434
+ paddd m3, m71435
+1436
+ psrad m0, INTERP_SHIFT_PP1437
+ psrad m1, INTERP_SHIFT_PP1438
+ psrad m2, INTERP_SHIFT_PP1439
+ psrad m3, INTERP_SHIFT_PP1440
+1441
+ packssdw m0, m11442
+ packssdw m2, m31443
+ pxor m5, m51444
+ CLIPW2 m0, m2, m5, m81445
+ movu [r2], xm01446
+ movu [r2 + r3], xm21447
+ vextracti32x4 [r2 + 2 * r3], m0, 11448
+ vextracti32x4 [r2 + r7], m2, 11449
+ lea r2, [r2 + 4 * r3]1450
+ vextracti32x4 [r2], m0, 21451
+ vextracti32x4 [r2 + r3], m2, 21452
+ vextracti32x4 [r2 + 2 * r3], m0, 31453
+ vextracti32x4 [r2 + r7], m2, 31454
+%endmacro1455
+1456
+;-----------------------------------------------------------------------------------------------------------------1457
+; void interp_4tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)1458
+;-----------------------------------------------------------------------------------------------------------------1459
+%if ARCH_X86_641460
+INIT_ZMM avx5121461
+cglobal interp_4tap_vert_pp_8x8, 5, 11, 91462
+ add r1d, r1d1463
+ add r3d, r3d1464
+ sub r0, r11465
+ shl r4d, 71466
+1467
+%ifdef PIC1468
+ lea r5, [tab_ChromaCoeffV_avx512]1469
+ lea r5, [r5 + r4]1470
+%else1471
+ lea r5, [tab_ChromaCoeffV_avx512 + r4]1472
+%endif1473
+ vbroadcasti32x8 m7, [INTERP_OFFSET_PP]1474
+ vbroadcasti32x8 m8, [pw_pixel_max]1475
+ lea r10, [3 * r1]1476
+ lea r7, [3 * r3]1477
+ PROCESS_CHROMA_VERT_PP_8x8_AVX5121478
+ RET1479
+%endif1480
+1481
+%macro FILTER_VER_PP_CHROMA_8xN_AVX512 11482
+INIT_ZMM avx5121483
+cglobal interp_4tap_vert_pp_8x%1, 5, 11, 91484
+ add r1d, r1d1485
+ add r3d, r3d1486
+ sub r0, r11487
+ shl r4d, 71488
+1489
+%ifdef PIC1490
+ lea r5, [tab_ChromaCoeffV_avx512]1491
+ lea r5, [r5 + r4]1492
+%else1493
+ lea r5, [tab_ChromaCoeffV_avx512 + r4]1494
+%endif1495
+ vbroadcasti32x8 m7, [INTERP_OFFSET_PP]1496
+ vbroadcasti32x8 m8, [pw_pixel_max]1497
+ lea r10, [3 * r1]1498
+ lea r7, [3 * r3]1499
+%rep %1/8 - 11500
+ PROCESS_CHROMA_VERT_PP_8x8_AVX5121501
+ lea r0, [r8 + 4 * r1]1502
+ lea r2, [r2 + 4 * r3]1503
+%endrep1504
+ PROCESS_CHROMA_VERT_PP_8x8_AVX5121505
+ RET1506
+%endmacro1507
+1508
+%if ARCH_X86_641509
+FILTER_VER_PP_CHROMA_8xN_AVX512 161510
+FILTER_VER_PP_CHROMA_8xN_AVX512 321511
+FILTER_VER_PP_CHROMA_8xN_AVX512 641512
+%endif1513
+1514
+%macro PROCESS_CHROMA_VERT_PP_16x4_AVX512 01515
+ movu ym1, [r0]1516
+ lea r6, [r0 + 2 * r1]1517
+ vinserti32x8 m1, [r6], 11518
+ movu ym3, [r0 + r1]1519
+ vinserti32x8 m3, [r6 + r1], 11520
+ punpcklwd m0, m1, m31521
+ pmaddwd m0, [r5]1522
+ punpckhwd m1, m31523
+ pmaddwd m1, [r5]1524
+1525
+ movu ym4, [r0 + 2 * r1]1526
+ vinserti32x8 m4, [r6 + 2 * r1], 11527
+ punpcklwd m2, m3, m41528
+ pmaddwd m2, [r5]1529
+ punpckhwd m3, m41530
+ pmaddwd m3, [r5]1531
+1532
+ lea r0, [r0 + 2 * r1]1533
+ lea r6, [r6 + 2 * r1]1534
+1535
+ movu ym5, [r0 + r1]1536
+ vinserti32x8 m5, [r6 + r1], 11537
+ punpcklwd m6, m4, m51538
+ pmaddwd m6, [r5 + mmsize]1539
+ paddd m0, m61540
+ punpckhwd m4, m51541
+ pmaddwd m4, [r5 + mmsize]1542
+ paddd m1, m41543
+1544
+ movu ym4, [r0 + 2 * r1]1545
+ vinserti32x8 m4, [r6 + 2 * r1], 11546
+ punpcklwd m6, m5, m41547
+ pmaddwd m6, [r5 + mmsize]1548
+ paddd m2, m61549
+ punpckhwd m5, m41550
+ pmaddwd m5, [r5 + mmsize]1551
+ paddd m3, m51552
+1553
+ paddd m0, m71554
+ paddd m1, m71555
+ paddd m2, m71556
+ paddd m3, m71557
+1558
+ psrad m0, INTERP_SHIFT_PP1559
+ psrad m1, INTERP_SHIFT_PP1560
+ psrad m2, INTERP_SHIFT_PP1561
+ psrad m3, INTERP_SHIFT_PP1562
+1563
+ packssdw m0, m11564
+ packssdw m2, m31565
+ pxor m5, m51566
+ CLIPW2 m0, m2, m5, m81567
+ movu [r2], ym01568
+ movu [r2 + r3], ym21569
+ vextracti32x8 [r2 + 2 * r3], m0, 11570
+ vextracti32x8 [r2 + r7], m2, 11571
+%endmacro1572
+1573
+;-----------------------------------------------------------------------------------------------------------------1574
+; void interp_4tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)1575
+;-----------------------------------------------------------------------------------------------------------------1576
+%if ARCH_X86_641577
+INIT_ZMM avx5121578
+cglobal interp_4tap_vert_pp_16x4, 5, 8, 91579
+ add r1d, r1d1580
+ add r3d, r3d1581
+ sub r0, r11582
+ shl r4d, 71583
+1584
+%ifdef PIC1585
+ lea r5, [tab_ChromaCoeffV_avx512]1586
+ lea r5, [r5 + r4]1587
+%else1588
+ lea r5, [tab_ChromaCoeffV_avx512 + r4]1589
+%endif1590
+ vbroadcasti32x8 m7, [INTERP_OFFSET_PP]1591
+ vbroadcasti32x8 m8, [pw_pixel_max]1592
+ lea r7, [3 * r3]1593
+ PROCESS_CHROMA_VERT_PP_16x4_AVX5121594
+ RET1595
+%endif1596
+1597
+%macro FILTER_VER_PP_CHROMA_16xN_AVX512 11598
+INIT_ZMM avx5121599
+cglobal interp_4tap_vert_pp_16x%1, 5, 8, 91600
+ add r1d, r1d1601
+ add r3d, r3d1602
+ sub r0, r11603
+ shl r4d, 71604
+1605
+%ifdef PIC1606
+ lea r5, [tab_ChromaCoeffV_avx512]1607
+ lea r5, [r5 + r4]1608
+%else1609
+ lea r5, [tab_ChromaCoeffV_avx512 + r4]1610
+%endif1611
+ vbroadcasti32x8 m7, [INTERP_OFFSET_PP]1612
+ vbroadcasti32x8 m8, [pw_pixel_max]1613
+ lea r7, [3 * r3]1614
+%rep %1/4 - 11615
+ PROCESS_CHROMA_VERT_PP_16x4_AVX5121616
+ lea r0, [r0 + 2 * r1]1617
+ lea r2, [r2 + 4 * r3]1618
+%endrep1619
+ PROCESS_CHROMA_VERT_PP_16x4_AVX5121620
+ RET1621
+%endmacro1622
+1623
+%if ARCH_X86_641624
+FILTER_VER_PP_CHROMA_16xN_AVX512 81625
+FILTER_VER_PP_CHROMA_16xN_AVX512 121626
+FILTER_VER_PP_CHROMA_16xN_AVX512 161627
+FILTER_VER_PP_CHROMA_16xN_AVX512 241628
+FILTER_VER_PP_CHROMA_16xN_AVX512 321629
+FILTER_VER_PP_CHROMA_16xN_AVX512 641630
+%endif1631
+1632
+%macro PROCESS_CHROMA_VERT_PP_24x8_AVX512 01633
+ movu ym1, [r0]1634
+ lea r6, [r0 + 2 * r1]1635
+ lea r8, [r0 + 4 * r1]1636
+ lea r9, [r8 + 2 * r1]1637
+1638
+ movu ym10, [r8]1639
+ movu ym3, [r0 + r1]1640
+ movu ym12, [r8 + r1]1641
+ vinserti32x8 m1, [r6], 11642
+ vinserti32x8 m10, [r9], 11643
+ vinserti32x8 m3, [r6 + r1], 11644
+ vinserti32x8 m12, [r9 + r1], 11645
+1646
+ punpcklwd m0, m1, m31647
+ punpcklwd m9, m10, m121648
+ pmaddwd m0, [r5]1649
+ pmaddwd m9, [r5]1650
+ punpckhwd m1, m31651
+ punpckhwd m10, m121652
+ pmaddwd m1, [r5]1653
+ pmaddwd m10, [r5]1654
+1655
+ movu ym4, [r0 + 2 * r1]1656
+ movu ym13, [r8 + 2 * r1]1657
+ vinserti32x8 m4, [r6 + 2 * r1], 11658
+ vinserti32x8 m13, [r9 + 2 * r1], 11659
+ punpcklwd m2, m3, m41660
+ punpcklwd m11, m12, m131661
+ pmaddwd m2, [r5]1662
+ pmaddwd m11, [r5]1663
+ punpckhwd m3, m41664
+ punpckhwd m12, m131665
+ pmaddwd m3, [r5]1666
+ pmaddwd m12, [r5]1667
+1668
+ movu ym5, [r0 + r10]1669
+ vinserti32x8 m5, [r6 + r10], 11670
+ movu ym14, [r8 + r10]1671
+ vinserti32x8 m14, [r9 + r10], 11672
+ punpcklwd m6, m4, m51673
+ punpcklwd m15, m13, m141674
+ pmaddwd m6, [r5 + mmsize]1675
+ pmaddwd m15, [r5 + mmsize]1676
+ paddd m0, m61677
+ paddd m9, m151678
+ punpckhwd m4, m51679
+ punpckhwd m13, m141680
+ pmaddwd m4, [r5 + mmsize]1681
+ pmaddwd m13, [r5 + mmsize]1682
+ paddd m1, m41683
+ paddd m10, m131684
+1685
+ movu ym4, [r0 + 4 * r1]1686
+ vinserti32x8 m4, [r6 + 4 * r1], 11687
+ movu ym13, [r8 + 4 * r1]1688
+ vinserti32x8 m13, [r9 + 4 * r1], 11689
+ punpcklwd m6, m5, m41690
+ punpcklwd m15, m14, m131691
+ pmaddwd m6, [r5 + mmsize]1692
+ pmaddwd m15, [r5 + mmsize]1693
+ paddd m2, m61694
+ paddd m11, m151695
+ punpckhwd m5, m41696
+ punpckhwd m14, m131697
+ pmaddwd m5, [r5 + mmsize]1698
+ pmaddwd m14, [r5 + mmsize]1699
+ paddd m3, m51700
+ paddd m12, m141701
+1702
+ paddd m0, m71703
+ paddd m1, m71704
+ paddd m2, m71705
+ paddd m3, m71706
+ paddd m9, m71707
+ paddd m10, m71708
+ paddd m11, m71709
+ paddd m12, m71710
+1711
+ psrad m0, INTERP_SHIFT_PP1712
+ psrad m1, INTERP_SHIFT_PP1713
+ psrad m2, INTERP_SHIFT_PP1714
+ psrad m3, INTERP_SHIFT_PP1715
+ psrad m9, INTERP_SHIFT_PP1716
+ psrad m10, INTERP_SHIFT_PP1717
+ psrad m11, INTERP_SHIFT_PP1718
+ psrad m12, INTERP_SHIFT_PP1719
+1720
+ packssdw m0, m11721
+ packssdw m2, m31722
+ packssdw m9, m101723
+ packssdw m11, m121724
+ pxor m5, m51725
+ CLIPW2 m0, m2, m5, m81726
+ CLIPW2 m9, m11, m5, m8 1727
+ movu [r2], ym01728
+ movu [r2 + r3], ym21729
+ vextracti32x8 [r2 + 2 * r3], m0, 11730
+ vextracti32x8 [r2 + r7], m2, 11731
+ lea r11, [r2 + 4 * r3]1732
+ movu [r11], ym91733
+ movu [r11 + r3], ym111734
+ vextracti32x8 [r11 + 2 * r3], m9, 11735
+ vextracti32x8 [r11 + r7], m11, 11736
+1737
+ movu xm1, [r0 + mmsize/2]1738
+ vinserti32x4 m1, [r6 + mmsize/2], 11739
+ vinserti32x4 m1, [r8 + mmsize/2], 21740
+ vinserti32x4 m1, [r9 + mmsize/2], 31741
+ movu xm3, [r0 + r1 + mmsize/2]1742
+ vinserti32x4 m3, [r6 + r1 + mmsize/2], 11743
+ vinserti32x4 m3, [r8 + r1 + mmsize/2], 21744
+ vinserti32x4 m3, [r9 + r1 + mmsize/2], 31745
+ punpcklwd m0, m1, m31746
+ pmaddwd m0, [r5]1747
+ punpckhwd m1, m31748
+ pmaddwd m1, [r5]1749
+1750
+ movu xm4, [r0 + 2 * r1 + mmsize/2]1751
+ vinserti32x4 m4, [r6 + 2 * r1 + mmsize/2], 11752
+ vinserti32x4 m4, [r8 + 2 * r1 + mmsize/2], 21753
+ vinserti32x4 m4, [r9 + 2 * r1 + mmsize/2], 31754
+ punpcklwd m2, m3, m41755
+ pmaddwd m2, [r5]1756
+ punpckhwd m3, m41757
+ pmaddwd m3, [r5]1758
+1759
+ movu xm5, [r0 + r10 + mmsize/2]1760
+ vinserti32x4 m5, [r6 + r10 + mmsize/2], 11761
+ vinserti32x4 m5, [r8 + r10 + mmsize/2], 21762
+ vinserti32x4 m5, [r9 + r10 + mmsize/2], 31763
+ punpcklwd m6, m4, m51764
+ pmaddwd m6, [r5 + mmsize]1765
+ paddd m0, m61766
+ punpckhwd m4, m51767
+ pmaddwd m4, [r5 + mmsize]1768
+ paddd m1, m41769
+1770
+ movu xm4, [r0 + 4 * r1 + mmsize/2]1771
+ vinserti32x4 m4, [r6 + 4 * r1 + mmsize/2], 11772
+ vinserti32x4 m4, [r8 + 4 * r1 + mmsize/2], 21773
+ vinserti32x4 m4, [r9 + 4 * r1 + mmsize/2], 31774
+ punpcklwd m6, m5, m41775
+ pmaddwd m6, [r5 + mmsize]1776
+ paddd m2, m61777
+ punpckhwd m5, m41778
+ pmaddwd m5, [r5 + mmsize]1779
+ paddd m3, m51780
+1781
+ paddd m0, m71782
+ paddd m1, m71783
+ paddd m2, m71784
+ paddd m3, m71785
+1786
+ psrad m0, INTERP_SHIFT_PP1787
+ psrad m1, INTERP_SHIFT_PP1788
+ psrad m2, INTERP_SHIFT_PP1789
+ psrad m3, INTERP_SHIFT_PP1790
+1791
+ packssdw m0, m11792
+ packssdw m2, m31793
+ pxor m5, m51794
+ CLIPW2 m0, m2, m5, m81795
+ movu [r2 + mmsize/2], xm01796
+ movu [r2 + r3 + mmsize/2], xm21797
+ vextracti32x4 [r2 + 2 * r3 + mmsize/2], m0, 11798
+ vextracti32x4 [r2 + r7 + mmsize/2], m2, 11799
+ lea r2, [r2 + 4 * r3]1800
+ vextracti32x4 [r2 + mmsize/2], m0, 21801
+ vextracti32x4 [r2 + r3 + mmsize/2], m2, 21802
+ vextracti32x4 [r2 + 2 * r3 + mmsize/2], m0, 31803
+ vextracti32x4 [r2 + r7 + mmsize/2], m2, 3 1804
+%endmacro1805
+1806
+%macro FILTER_VER_PP_CHROMA_24xN_AVX512 11807
+INIT_ZMM avx5121808
+cglobal interp_4tap_vert_pp_24x%1, 5, 12, 161809
+ add r1d, r1d1810
+ add r3d, r3d1811
+ sub r0, r11812
+ shl r4d, 71813
+1814
+%ifdef PIC1815
+ lea r5, [tab_ChromaCoeffV_avx512]1816
+ lea r5, [r5 + r4]1817
+%else1818
+ lea r5, [tab_ChromaCoeffV_avx512 + r4]1819
+%endif1820
+ vbroadcasti32x8 m7, [INTERP_OFFSET_PP]1821
+ vbroadcasti32x8 m8, [pw_pixel_max]1822
+ lea r10, [3 * r1]1823
+ lea r7, [3 * r3]1824
+%rep %1/8 - 11825
+ PROCESS_CHROMA_VERT_PP_24x8_AVX5121826
+ lea r0, [r8 + 4 * r1]1827
+ lea r2, [r2 + 4 * r3]1828
+%endrep1829
+ PROCESS_CHROMA_VERT_PP_24x8_AVX5121830
+ RET1831
+%endmacro1832
+1833
+%if ARCH_X86_641834
+ FILTER_VER_PP_CHROMA_24xN_AVX512 321835
+ FILTER_VER_PP_CHROMA_24xN_AVX512 641836
+%endif1837
+1838
+%macro PROCESS_CHROMA_VERT_PP_32x2_AVX512 01839
+ movu m1, [r0]1840
+ movu m3, [r0 + r1]1841
+ punpcklwd m0, m1, m31842
+ pmaddwd m0, [r5]1843
+ punpckhwd m1, m31844
+ pmaddwd m1, [r5]1845
+1846
+ movu m4, [r0 + 2 * r1]1847
+ punpcklwd m2, m3, m41848
+ pmaddwd m2, [r5]1849
+ punpckhwd m3, m41850
+ pmaddwd m3, [r5]1851
+1852
+ lea r0, [r0 + 2 * r1]1853
+ movu m5, [r0 + r1]1854
+ punpcklwd m6, m4, m51855
+ pmaddwd m6, [r5 + mmsize]1856
+ paddd m0, m61857
+ punpckhwd m4, m51858
+ pmaddwd m4, [r5 + mmsize]1859
+ paddd m1, m41860
+1861
+ movu m4, [r0 + 2 * r1]1862
+ punpcklwd m6, m5, m41863
+ pmaddwd m6, [r5 + mmsize]1864
+ paddd m2, m61865
+ punpckhwd m5, m41866
+ pmaddwd m5, [r5 + mmsize]1867
+ paddd m3, m51868
+1869
+ paddd m0, m71870
+ paddd m1, m71871
+ paddd m2, m71872
+ paddd m3, m71873
+1874
+ psrad m0, INTERP_SHIFT_PP1875
+ psrad m1, INTERP_SHIFT_PP1876
+ psrad m2, INTERP_SHIFT_PP1877
+ psrad m3, INTERP_SHIFT_PP1878
+1879
+ packssdw m0, m11880
+ packssdw m2, m31881
+ pxor m5, m51882
+ CLIPW2 m0, m2, m5, m81883
+ movu [r2], m01884
+ movu [r2 + r3], m21885
+%endmacro1886
+1887
+;-----------------------------------------------------------------------------------------------------------------1888
+; void interp_4tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)1889
+;-----------------------------------------------------------------------------------------------------------------1890
+%macro FILTER_VER_PP_CHROMA_32xN_AVX512 11891
+INIT_ZMM avx5121892
+cglobal interp_4tap_vert_pp_32x%1, 5, 7, 91893
+ add r1d, r1d1894
+ add r3d, r3d1895
+ sub r0, r11896
+ shl r4d, 71897
+1898
+%ifdef PIC1899
+ lea r5, [tab_ChromaCoeffV_avx512]1900
+ lea r5, [r5 + r4]1901
+%else1902
+ lea r5, [tab_ChromaCoeffV_avx512 + r4]1903
+%endif1904
+ vbroadcasti32x8 m7, [INTERP_OFFSET_PP]1905
+ vbroadcasti32x8 m8, [pw_pixel_max]1906
+1907
+%rep %1/2 - 11908
+ PROCESS_CHROMA_VERT_PP_32x2_AVX5121909
+ lea r2, [r2 + 2 * r3]1910
+%endrep1911
+ PROCESS_CHROMA_VERT_PP_32x2_AVX5121912
+ RET1913
+%endmacro1914
+1915
+%if ARCH_X86_641916
+FILTER_VER_PP_CHROMA_32xN_AVX512 81917
+FILTER_VER_PP_CHROMA_32xN_AVX512 161918
+FILTER_VER_PP_CHROMA_32xN_AVX512 241919
+FILTER_VER_PP_CHROMA_32xN_AVX512 321920
+FILTER_VER_PP_CHROMA_32xN_AVX512 481921
+FILTER_VER_PP_CHROMA_32xN_AVX512 641922
+%endif1923
+1924
+%macro PROCESS_CHROMA_VERT_PP_48x4_AVX512 01925
+ movu m1, [r0]1926
+ lea r6, [r0 + 2 * r1]1927
+ movu m10, [r6]1928
+ movu m3, [r0 + r1]1929
+ movu m12, [r6 + r1]1930
+ punpcklwd m0, m1, m31931
+ punpcklwd m9, m10, m121932
+ pmaddwd m0, [r5]1933
+ pmaddwd m9, [r5]1934
+ punpckhwd m1, m31935
+ punpckhwd m10, m121936
+ pmaddwd m1, [r5]1937
+ pmaddwd m10, [r5]1938
+1939
+ movu m4, [r0 + 2 * r1]1940
+ movu m13, [r6 + 2 * r1]1941
+ punpcklwd m2, m3, m41942
+ punpcklwd m11, m12, m131943
+ pmaddwd m2, [r5]1944
+ pmaddwd m11, [r5]1945
+ punpckhwd m3, m41946
+ punpckhwd m12, m131947
+ pmaddwd m3, [r5]1948
+ pmaddwd m12, [r5]1949
+1950
+ movu m5, [r0 + r7]1951
+ movu m14, [r6 + r7]1952
+ punpcklwd m6, m4, m51953
+ punpcklwd m15, m13, m141954
+ pmaddwd m6, [r5 + mmsize]1955
+ pmaddwd m15, [r5 + mmsize]1956
+ paddd m0, m61957
+ paddd m9, m151958
+ punpckhwd m4, m51959
+ punpckhwd m13, m141960
+ pmaddwd m4, [r5 + mmsize]1961
+ pmaddwd m13, [r5 + mmsize]1962
+ paddd m1, m41963
+ paddd m10, m131964
+1965
+ movu m4, [r0 + 4 * r1]1966
+ movu m13, [r6 + 4 * r1]1967
+ punpcklwd m6, m5, m41968
+ punpcklwd m15, m14, m131969
+ pmaddwd m6, [r5 + mmsize]1970
+ pmaddwd m15, [r5 + mmsize]1971
+ paddd m2, m61972
+ paddd m11, m151973
+ punpckhwd m5, m41974
+ punpckhwd m14, m131975
+ pmaddwd m5, [r5 + mmsize]1976
+ pmaddwd m14, [r5 + mmsize]1977
+ paddd m3, m51978
+ paddd m12, m141979
+1980
+ paddd m0, m71981
+ paddd m1, m71982
+ paddd m2, m71983
+ paddd m3, m71984
+ paddd m9, m71985
+ paddd m10, m71986
+ paddd m11, m71987
+ paddd m12, m71988
+1989
+ psrad m0, INTERP_SHIFT_PP1990
+ psrad m1, INTERP_SHIFT_PP1991
+ psrad m2, INTERP_SHIFT_PP1992
+ psrad m3, INTERP_SHIFT_PP1993
+ psrad m9, INTERP_SHIFT_PP1994
+ psrad m10, INTERP_SHIFT_PP1995
+ psrad m11, INTERP_SHIFT_PP1996
+ psrad m12, INTERP_SHIFT_PP1997
+1998
+ packssdw m0, m11999
+ packssdw m2, m32000
+ packssdw m9, m102001
+ packssdw m11, m122002
+ CLIPW2 m0, m2, m16, m82003
+ CLIPW2 m9, m11, m16, m82004
+ movu [r2], m02005
+ movu [r2 + r3], m22006
+ movu [r2 + 2 * r3], m92007
+ movu [r2 + r8], m112008
+2009
+ movu ym1, [r0 + mmsize]2010
+ vinserti32x8 m1, [r6 + mmsize], 12011
+ movu ym3, [r0 + r1 + mmsize]2012
+ vinserti32x8 m3, [r6 + r1 + mmsize], 12013
+ punpcklwd m0, m1, m32014
+ pmaddwd m0, [r5]2015
+ punpckhwd m1, m32016
+ pmaddwd m1, [r5]2017
+2018
+ movu ym4, [r0 + 2 * r1 + mmsize]2019
+ vinserti32x8 m4, [r6 + 2 * r1 + mmsize], 12020
+ punpcklwd m2, m3, m42021
+ pmaddwd m2, [r5]2022
+ punpckhwd m3, m42023
+ pmaddwd m3, [r5]2024
+2025
+ movu ym5, [r0 + r7 + mmsize]2026
+ vinserti32x8 m5, [r6 + r7 + mmsize], 12027
+ punpcklwd m6, m4, m52028
+ pmaddwd m6, [r5 + mmsize]2029
+ paddd m0, m62030
+ punpckhwd m4, m52031
+ pmaddwd m4, [r5 + mmsize]2032
+ paddd m1, m42033
+2034
+ movu ym4, [r0 + 4 * r1 + mmsize]2035
+ vinserti32x8 m4, [r6 + 4 * r1 + mmsize], 12036
+ punpcklwd m6, m5, m42037
+ pmaddwd m6, [r5 + mmsize]2038
+ paddd m2, m62039
+ punpckhwd m5, m42040
+ pmaddwd m5, [r5 + mmsize]2041
+ paddd m3, m52042
+2043
+ paddd m0, m72044
+ paddd m1, m72045
+ paddd m2, m72046
+ paddd m3, m72047
+2048
+ psrad m0, INTERP_SHIFT_PP2049
+ psrad m1, INTERP_SHIFT_PP2050
+ psrad m2, INTERP_SHIFT_PP2051
+ psrad m3, INTERP_SHIFT_PP2052
+2053
+ packssdw m0, m12054
+ packssdw m2, m32055
+ CLIPW2 m0, m2, m16, m82056
+ movu [r2 + mmsize], ym02057
+ movu [r2 + r3 + mmsize], ym22058
+ vextracti32x8 [r2 + 2 * r3 + mmsize], m0, 12059
+ vextracti32x8 [r2 + r8 + mmsize], m2, 12060
+%endmacro2061
+2062
+%if ARCH_X86_642063
+INIT_ZMM avx5122064
+cglobal interp_4tap_vert_pp_48x64, 5, 9, 172065
+ add r1d, r1d2066
+ add r3d, r3d2067
+ sub r0, r12068
+ shl r4d, 72069
+%ifdef PIC2070
+ lea r5, [tab_ChromaCoeffV_avx512]2071
+ lea r5, [r5 + r4]2072
+%else2073
+ lea r5, [tab_ChromaCoeffV_avx512 + r4]2074
+%endif2075
+ lea r7, [3 * r1]2076
+ lea r8, [3 * r3]2077
+ vbroadcasti32x8 m7, [INTERP_OFFSET_PP]2078
+ vbroadcasti32x8 m8, [pw_pixel_max]2079
+ pxor m16, m162080
+2081
+%rep 152082
+ PROCESS_CHROMA_VERT_PP_48x4_AVX5122083
+ lea r0, [r0 + 4 * r1]2084
+ lea r2, [r2 + 4 * r3]2085
+%endrep2086
+ PROCESS_CHROMA_VERT_PP_48x4_AVX5122087
+ RET2088
+%endif2089
+2090
+%macro PROCESS_CHROMA_VERT_PP_64x2_AVX512 02091
+ movu m1, [r0]2092
+ movu m3, [r0 + r1]2093
+ punpcklwd m0, m1, m32094
+ pmaddwd m0, [r5]2095
+ punpckhwd m1, m32096
+ pmaddwd m1, [r5]2097
+2098
+ movu m9, [r0 + mmsize]2099
+ movu m11, [r0 + r1 + mmsize]2100
+ punpcklwd m8, m9, m112101
+ pmaddwd m8, [r5]2102
+ punpckhwd m9, m112103
+ pmaddwd m9, [r5]2104
+2105
+ movu m4, [r0 + 2 * r1]2106
+ punpcklwd m2, m3, m42107
+ pmaddwd m2, [r5]2108
+ punpckhwd m3, m42109
+ pmaddwd m3, [r5]2110
+2111
+ movu m12, [r0 + 2 * r1 + mmsize]2112
+ punpcklwd m10, m11, m122113
+ pmaddwd m10, [r5]2114
+ punpckhwd m11, m122115
+ pmaddwd m11, [r5]2116
+2117
+ lea r0, [r0 + 2 * r1]2118
+ movu m5, [r0 + r1]2119
+ punpcklwd m6, m4, m52120
+ pmaddwd m6, [r5 + 1 * mmsize]2121
+ paddd m0, m62122
+ punpckhwd m4, m52123
+ pmaddwd m4, [r5 + 1 * mmsize]2124
+ paddd m1, m42125
+2126
+ movu m13, [r0 + r1 + mmsize]2127
+ punpcklwd m14, m12, m132128
+ pmaddwd m14, [r5 + 1 * mmsize]2129
+ paddd m8, m142130
+ punpckhwd m12, m132131
+ pmaddwd m12, [r5 + 1 * mmsize]2132
+ paddd m9, m122133
+2134
+ movu m4, [r0 + 2 * r1]2135
+ punpcklwd m6, m5, m42136
+ pmaddwd m6, [r5 + 1 * mmsize]2137
+ paddd m2, m62138
+ punpckhwd m5, m42139
+ pmaddwd m5, [r5 + 1 * mmsize]2140
+ paddd m3, m52141
+2142
+ movu m12, [r0 + 2 * r1 + mmsize]2143
+ punpcklwd m14, m13, m122144
+ pmaddwd m14, [r5 + 1 * mmsize]2145
+ paddd m10, m142146
+ punpckhwd m13, m122147
+ pmaddwd m13, [r5 + 1 * mmsize]2148
+ paddd m11, m132149
+2150
+ paddd m0, m72151
+ paddd m1, m72152
+ paddd m2, m72153
+ paddd m3, m72154
+ paddd m8, m72155
+ paddd m9, m72156
+ paddd m10, m72157
+ paddd m11, m72158
+2159
+ psrad m0, INTERP_SHIFT_PP2160
+ psrad m1, INTERP_SHIFT_PP2161
+ psrad m2, INTERP_SHIFT_PP2162
+ psrad m3, INTERP_SHIFT_PP2163
+ psrad m8, INTERP_SHIFT_PP2164
+ psrad m9, INTERP_SHIFT_PP2165
+ psrad m10, INTERP_SHIFT_PP2166
+ psrad m11, INTERP_SHIFT_PP2167
+2168
+ packssdw m0, m12169
+ packssdw m2, m32170
+ packssdw m8, m92171
+ packssdw m10, m112172
+ pxor m5, m52173
+ CLIPW2 m0, m2, m5, m152174
+ CLIPW2 m8, m10, m5, m152175
+ movu [r2], m02176
+ movu [r2 + r3], m22177
+ movu [r2 + mmsize], m82178
+ movu [r2 + r3 + mmsize], m102179
+%endmacro2180
+2181
+;-----------------------------------------------------------------------------------------------------------------2182
+; void interp_4tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)2183
+;-----------------------------------------------------------------------------------------------------------------2184
+%macro FILTER_VER_PP_CHROMA_64xN_AVX512 12185
+INIT_ZMM avx5122186
+cglobal interp_4tap_vert_pp_64x%1, 5, 7, 162187
+ add r1d, r1d2188
+ add r3d, r3d2189
+ sub r0, r12190
+ shl r4d, 72191
+2192
+%ifdef PIC2193
+ lea r5, [tab_ChromaCoeffV_avx512]2194
+ lea r5, [r5 + r4]2195
+%else2196
+ lea r5, [tab_ChromaCoeffV_avx512 + r4]2197
+%endif2198
+ vbroadcasti32x8 m7, [INTERP_OFFSET_PP]2199
+ vbroadcasti32x8 m15, [pw_pixel_max]2200
+2201
+%rep %1/2 - 12202
+ PROCESS_CHROMA_VERT_PP_64x2_AVX5122203
+ lea r2, [r2 + 2 * r3]2204
+%endrep2205
+ PROCESS_CHROMA_VERT_PP_64x2_AVX5122206
+ RET2207
+%endmacro2208
+2209
+%if ARCH_X86_642210
+FILTER_VER_PP_CHROMA_64xN_AVX512 162211
+FILTER_VER_PP_CHROMA_64xN_AVX512 322212
+FILTER_VER_PP_CHROMA_64xN_AVX512 482213
+FILTER_VER_PP_CHROMA_64xN_AVX512 642214
+%endif2215
+;-------------------------------------------------------------------------------------------------------------2216
+; avx512 chroma_vpp code end2217
+;-------------------------------------------------------------------------------------------------------------2218
+;-------------------------------------------------------------------------------------------------------------2219
+; avx512 chroma_hps code start2220
+;-------------------------------------------------------------------------------------------------------------2221
+%macro PROCESS_IPFILTER_CHROMA_PS_32x2_AVX512 02222
+ ; register map2223
+ ; m0 , m1 - interpolate coeff2224
+ ; m2 , m3 - shuffle load order table2225
+ ; m4 - INTERP_OFFSET_PS2226
+ ; m5 - shuffle store order table2227
+2228
+ movu m6, [r0]2229
+ movu m7, [r0 + 8]2230
+2231
+ pshufb m8, m6, m32232
+ pshufb m6, m22233
+ pmaddwd m6, m02234
+ pmaddwd m8, m12235
+ paddd m6, m82236
+ paddd m6, m42237
+ psrad m6, INTERP_SHIFT_PS2238
+2239
+ pshufb m8, m7, m32240
+ pshufb m7, m22241
+ pmaddwd m7, m02242
+ pmaddwd m8, m12243
+ paddd m7, m82244
+ paddd m7, m42245
+ psrad m7, INTERP_SHIFT_PS2246
+2247
+ packssdw m6, m72248
+ pshufb m6, m52249
+ movu [r2], m62250
+2251
+ movu m6, [r0 + r1]2252
+ movu m7, [r0 + r1 + 8]2253
+2254
+ pshufb m8, m6, m32255
+ pshufb m6, m22256
+ pmaddwd m6, m02257
+ pmaddwd m8, m12258
+ paddd m6, m82259
+ paddd m6, m42260
+ psrad m6, INTERP_SHIFT_PS2261
+2262
+ pshufb m8, m7, m32263
+ pshufb m7, m22264
+ pmaddwd m7, m02265
+ pmaddwd m8, m12266
+ paddd m7, m82267
+ paddd m7, m42268
+ psrad m7, INTERP_SHIFT_PS2269
+2270
+ packssdw m6, m72271
+ pshufb m6, m52272
+ movu [r2 + r3], m62273
+%endmacro2274
+2275
+%macro PROCESS_IPFILTER_CHROMA_PS_32x1_AVX512 02276
+ movu m6, [r0]2277
+ movu m7, [r0 + 8]2278
+2279
+ pshufb m8, m6, m32280
+ pshufb m6, m22281
+ pmaddwd m6, m02282
+ pmaddwd m8, m12283
+ paddd m6, m82284
+ paddd m6, m42285
+ psrad m6, INTERP_SHIFT_PS2286
+2287
+ pshufb m8, m7, m32288
+ pshufb m7, m22289
+ pmaddwd m7, m02290
+ pmaddwd m8, m12291
+ paddd m7, m82292
+ paddd m7, m42293
+ psrad m7, INTERP_SHIFT_PS2294
+2295
+ packssdw m6, m72296
+ pshufb m6, m52297
+ movu [r2], m62298
+%endmacro2299
+2300
+%macro IPFILTER_CHROMA_PS_AVX512_32xN 12301
+%if ARCH_X86_64 == 12302
+INIT_ZMM avx5122303
+cglobal interp_4tap_horiz_ps_32x%1, 4,7,92304
+ shl r1d, 12305
+ shl r3d, 12306
+ mov r4d, r4m2307
+ mov r5d, r5m2308
+%ifdef PIC2309
+ lea r6, [tab_ChromaCoeff]2310
+ vpbroadcastd m0, [r6 + r4 * 8]2311
+ vpbroadcastd m1, [r6 + r4 * 8 + 4]2312
+%else2313
+ vpbroadcastd m0, [tab_ChromaCoeff + r4 * 8]2314
+ vpbroadcastd m1, [tab_ChromaCoeff + r4 * 8 + 4]2315
+%endif2316
+ vbroadcasti32x8 m2, [interp8_hpp_shuf1_load_avx512]2317
+ vbroadcasti32x8 m3, [interp8_hpp_shuf2_load_avx512]2318
+ vbroadcasti32x4 m4, [INTERP_OFFSET_PS]2319
+ vbroadcasti32x8 m5, [interp8_hpp_shuf1_store_avx512]2320
+2321
+ mov r6d, %12322
+ sub r0, 22323
+ test r5d, r5d2324
+ jz .loop2325
+ sub r0, r12326
+ add r6d, 32327
+ PROCESS_IPFILTER_CHROMA_PS_32x1_AVX5122328
+ lea r0, [r0 + r1]2329
+ lea r2, [r2 + r3]2330
+ dec r6d2331
+2332
+.loop:2333
+ PROCESS_IPFILTER_CHROMA_PS_32x2_AVX5122334
+ lea r0, [r0 + 2 * r1]2335
+ lea r2, [r2 + 2 * r3]2336
+ sub r6d, 22337
+ jnz .loop2338
+ RET2339
+%endif2340
+%endmacro2341
+2342
+IPFILTER_CHROMA_PS_AVX512_32xN 82343
+IPFILTER_CHROMA_PS_AVX512_32xN 162344
+IPFILTER_CHROMA_PS_AVX512_32xN 242345
+IPFILTER_CHROMA_PS_AVX512_32xN 322346
+IPFILTER_CHROMA_PS_AVX512_32xN 482347
+IPFILTER_CHROMA_PS_AVX512_32xN 642348
+2349
+%macro PROCESS_IPFILTER_CHROMA_PS_64x2_AVX512 02350
+ ; register map2351
+ ; m0 , m1 - interpolate coeff2352
+ ; m2 , m3 -shuffle order table2353
+ ; m4 - INTERP_OFFSET_PS2354
+ ; m5 - shuffle store order table2355
+2356
+2357
+ movu m6, [r0]2358
+ movu m7, [r0 + 8]2359
+2360
+ pshufb m8, m6, m32361
+ pshufb m6, m22362
+ pmaddwd m6, m02363
+ pmaddwd m8, m12364
+ paddd m6, m82365
+ paddd m6, m42366
+ psrad m6, INTERP_SHIFT_PS2367
+2368
+ pshufb m8, m7, m32369
+ pshufb m7, m22370
+ pmaddwd m7, m02371
+ pmaddwd m8, m12372
+ paddd m7, m82373
+ paddd m7, m42374
+ psrad m7, INTERP_SHIFT_PS2375
+2376
+ packssdw m6, m72377
+ pshufb m6, m52378
+ movu [r2], m62379
+2380
+ movu m6, [r0 + mmsize]2381
+ movu m7, [r0 + mmsize + 8]2382
+2383
+ pshufb m8, m6, m32384
+ pshufb m6, m22385
+ pmaddwd m6, m02386
+ pmaddwd m8, m12387
+ paddd m6, m82388
+ paddd m6, m42389
+ psrad m6, INTERP_SHIFT_PS2390
+2391
+ pshufb m8, m7, m32392
+ pshufb m7, m22393
+ pmaddwd m7, m02394
+ pmaddwd m8, m12395
+ paddd m7, m82396
+ paddd m7, m42397
+ psrad m7, INTERP_SHIFT_PS2398
+2399
+ packssdw m6, m72400
+ pshufb m6, m52401
+ movu [r2 + mmsize], m62402
+2403
+ movu m6, [r0 + r1]2404
+ movu m7, [r0 + r1 + 8]2405
+2406
+ pshufb m8, m6, m32407
+ pshufb m6, m22408
+ pmaddwd m6, m02409
+ pmaddwd m8, m12410
+ paddd m6, m82411
+ paddd m6, m42412
+ psrad m6, INTERP_SHIFT_PS2413
+2414
+ pshufb m8, m7, m32415
+ pshufb m7, m22416
+ pmaddwd m7, m02417
+ pmaddwd m8, m12418
+ paddd m7, m82419
+ paddd m7, m42420
+ psrad m7, INTERP_SHIFT_PS2421
+2422
+ packssdw m6, m72423
+ pshufb m6, m52424
+ movu [r2 + r3], m62425
+2426
+ movu m6, [r0 + r1 + mmsize]2427
+ movu m7, [r0 + r1 + mmsize + 8]2428
+2429
+ pshufb m8, m6, m32430
+ pshufb m6, m22431
+ pmaddwd m6, m02432
+ pmaddwd m8, m12433
+ paddd m6, m82434
+ paddd m6, m42435
+ psrad m6, INTERP_SHIFT_PS2436
+2437
+ pshufb m8, m7, m32438
+ pshufb m7, m22439
+ pmaddwd m7, m02440
+ pmaddwd m8, m12441
+ paddd m7, m82442
+ paddd m7, m42443
+ psrad m7, INTERP_SHIFT_PS2444
+2445
+ packssdw m6, m72446
+ pshufb m6, m52447
+ movu [r2 + r3 + mmsize], m62448
+%endmacro2449
+2450
+%macro PROCESS_IPFILTER_CHROMA_PS_64x1_AVX512 02451
+ movu m6, [r0]2452
+ movu m7, [r0 + 8]2453
+2454
+ pshufb m8, m6, m32455
+ pshufb m6, m22456
+ pmaddwd m6, m02457
+ pmaddwd m8, m12458
+ paddd m6, m82459
+ paddd m6, m42460
+ psrad m6, INTERP_SHIFT_PS2461
+2462
+ pshufb m8, m7, m32463
+ pshufb m7, m22464
+ pmaddwd m7, m02465
+ pmaddwd m8, m12466
+ paddd m7, m82467
+ paddd m7, m42468
+ psrad m7, INTERP_SHIFT_PS2469
+2470
+ packssdw m6, m72471
+ pshufb m6, m52472
+ movu [r2], m62473
+2474
+ movu m6, [r0 + mmsize]2475
+ movu m7, [r0 + mmsize + 8]2476
+2477
+ pshufb m8, m6, m32478
+ pshufb m6, m22479
+ pmaddwd m6, m02480
+ pmaddwd m8, m12481
+ paddd m6, m82482
+ paddd m6, m42483
+ psrad m6, INTERP_SHIFT_PS2484
+2485
+ pshufb m8, m7, m32486
+ pshufb m7, m22487
+ pmaddwd m7, m02488
+ pmaddwd m8, m12489
+ paddd m7, m82490
+ paddd m7, m42491
+ psrad m7, INTERP_SHIFT_PS2492
+2493
+ packssdw m6, m72494
+ pshufb m6, m52495
+ movu [r2 + mmsize], m62496
+%endmacro2497
+2498
+%macro IPFILTER_CHROMA_PS_AVX512_64xN 12499
+%if ARCH_X86_64 == 12500
+INIT_ZMM avx5122501
+cglobal interp_4tap_horiz_ps_64x%1, 4,7,92502
+ shl r1d, 12503
+ shl r3d, 12504
+ mov r4d, r4m2505
+ mov r5d, r5m2506
+%ifdef PIC2507
+ lea r6, [tab_ChromaCoeff]2508
+ vpbroadcastd m0, [r6 + r4 * 8]2509
+ vpbroadcastd m1, [r6 + r4 * 8 + 4]2510
+%else2511
+ vpbroadcastd m0, [tab_ChromaCoeff + r4 * 8]2512
+ vpbroadcastd m1, [tab_ChromaCoeff + r4 * 8 + 4]2513
+%endif2514
+ vbroadcasti32x8 m2, [interp8_hpp_shuf1_load_avx512]2515
+ vbroadcasti32x8 m3, [interp8_hpp_shuf2_load_avx512]2516
+ vbroadcasti32x4 m4, [INTERP_OFFSET_PS]2517
+ vbroadcasti32x8 m5, [interp8_hpp_shuf1_store_avx512]2518
+ mov r6d, %12519
+ sub r0, 22520
+ test r5d, r5d2521
+ jz .loop2522
+ sub r0, r12523
+ add r6d, 32524
+ PROCESS_IPFILTER_CHROMA_PS_64x1_AVX5122525
+ lea r0, [r0 + r1]2526
+ lea r2, [r2 + r3]2527
+ dec r6d2528
+2529
+.loop:2530
+ PROCESS_IPFILTER_CHROMA_PS_64x2_AVX5122531
+ lea r0, [r0 + 2 * r1]2532
+ lea r2, [r2 + 2 * r3]2533
+ sub r6d, 22534
+ jnz .loop2535
+ RET2536
+%endif2537
+%endmacro2538
+2539
+IPFILTER_CHROMA_PS_AVX512_64xN 162540
+IPFILTER_CHROMA_PS_AVX512_64xN 322541
+IPFILTER_CHROMA_PS_AVX512_64xN 482542
+IPFILTER_CHROMA_PS_AVX512_64xN 642543
+2544
+%macro PROCESS_IPFILTER_CHROMA_PS_16x2_AVX512 02545
+ ; register map2546
+ ; m0 , m1 - interpolate coeff2547
+ ; m2 , m3 - shuffle order table2548
+ ; m4 - INTERP_OFFSET_PS2549
+ ; m5 - shuffle store order table2550
+2551
+ movu ym6, [r0]2552
+ vinserti32x8 m6, [r0 + r1], 12553
+ movu ym7, [r0 + 8]2554
+ vinserti32x8 m7, [r0 + r1 + 8], 12555
+2556
+ pshufb m8, m6, m32557
+ pshufb m6, m22558
+ pmaddwd m6, m02559
+ pmaddwd m8, m12560
+ paddd m6, m82561
+ paddd m6, m42562
+ psrad m6, INTERP_SHIFT_PS2563
+2564
+ pshufb m8, m7, m32565
+ pshufb m7, m22566
+ pmaddwd m7, m02567
+ pmaddwd m8, m12568
+ paddd m7, m82569
+ paddd m7, m42570
+ psrad m7, INTERP_SHIFT_PS2571
+2572
+ packssdw m6, m72573
+ pshufb m6, m52574
+ movu [r2], ym62575
+ vextracti32x8 [r2 + r3], m6, 12576
+%endmacro2577
+%macro PROCESS_IPFILTER_CHROMA_PS_16x1_AVX512 02578
+ movu ym6, [r0]2579
+ vinserti32x8 m6, [r0 + 8], 12580
+2581
+ pshufb m8, m6, m32582
+ pshufb m6, m22583
+ pmaddwd m6, m02584
+ pmaddwd m8, m12585
+ paddd m6, m82586
+ paddd m6, m42587
+ psrad m6, INTERP_SHIFT_PS2588
+2589
+ vextracti32x8 ym7, m6, 12590
+ packssdw ym6, ym72591
+ pshufb ym6, ym52592
+ movu [r2], ym62593
+%endmacro2594
+%macro IPFILTER_CHROMA_PS_AVX512_16xN 12595
+%if ARCH_X86_64 == 12596
+INIT_ZMM avx5122597
+cglobal interp_4tap_horiz_ps_16x%1, 4,7,92598
+ shl r1d, 12599
+ shl r3d, 12600
+ mov r4d, r4m2601
+ mov r5d, r5m2602
+%ifdef PIC2603
+ lea r6, [tab_ChromaCoeff]2604
+ vpbroadcastd m0, [r6 + r4 * 8]2605
+ vpbroadcastd m1, [r6 + r4 * 8 + 4]2606
+%else2607
+ vpbroadcastd m0, [tab_ChromaCoeff + r4 * 8]2608
+ vpbroadcastd m1, [tab_ChromaCoeff + r4 * 8 + 4]2609
+%endif2610
+ mova m2, [interp8_hpp_shuf1_load_avx512]2611
+ mova m3, [interp8_hpp_shuf2_load_avx512]2612
+ vbroadcasti32x4 m4, [INTERP_OFFSET_PS]2613
+ mova m5, [interp8_hpp_shuf1_store_avx512]2614
+ mov r6d, %12615
+ sub r0, 22616
+ test r5d, r5d2617
+ jz .loop2618
+ sub r0, r12619
+ add r6d, 32620
+ PROCESS_IPFILTER_CHROMA_PS_16x1_AVX5122621
+ lea r0, [r0 + r1]2622
+ lea r2, [r2 + r3]2623
+ dec r6d2624
+2625
+.loop:2626
+ PROCESS_IPFILTER_CHROMA_PS_16x2_AVX5122627
+ lea r0, [r0 + 2 * r1]2628
+ lea r2, [r2 + 2 * r3]2629
+ sub r6d, 22630
+ jnz .loop2631
+ RET2632
+%endif2633
+%endmacro2634
+2635
+IPFILTER_CHROMA_PS_AVX512_16xN 42636
+IPFILTER_CHROMA_PS_AVX512_16xN 82637
+IPFILTER_CHROMA_PS_AVX512_16xN 122638
+IPFILTER_CHROMA_PS_AVX512_16xN 162639
+IPFILTER_CHROMA_PS_AVX512_16xN 242640
+IPFILTER_CHROMA_PS_AVX512_16xN 322641
+IPFILTER_CHROMA_PS_AVX512_16xN 642642
+2643
+%macro PROCESS_IPFILTER_CHROMA_PS_48x2_AVX512 02644
+ ; register map2645
+ ; m0 , m1 - interpolate coeff2646
+ ; m2 , m3 - shuffle load order table2647
+ ; m4 - INTERP_OFFSET_PS2648
+ ; m5 - shuffle store order table2649
+2650
+ movu m6, [r0]2651
+ movu m7, [r0 + 8]2652
+2653
+ pshufb m8, m6, m32654
+ pshufb m6, m22655
+ pmaddwd m6, m02656
+ pmaddwd m8, m12657
+ paddd m6, m82658
+ paddd m6, m42659
+ psrad m6, INTERP_SHIFT_PS2660
+2661
+ pshufb m8, m7, m32662
+ pshufb m7, m22663
+ pmaddwd m7, m02664
+ pmaddwd m8, m12665
+ paddd m7, m82666
+ paddd m7, m42667
+ psrad m7, INTERP_SHIFT_PS2668
+2669
+ packssdw m6, m72670
+ pshufb m6, m52671
+ movu [r2], m62672
+2673
+ movu m6, [r0 + r1]2674
+ movu m7, [r0 + r1 + 8]2675
+2676
+ pshufb m8, m6, m32677
+ pshufb m6, m22678
+ pmaddwd m6, m02679
+ pmaddwd m8, m12680
+ paddd m6, m82681
+ paddd m6, m42682
+ psrad m6, INTERP_SHIFT_PS2683
+2684
+ pshufb m8, m7, m32685
+ pshufb m7, m22686
+ pmaddwd m7, m02687
+ pmaddwd m8, m12688
+ paddd m7, m82689
+ paddd m7, m42690
+ psrad m7, INTERP_SHIFT_PS2691
+2692
+ packssdw m6, m72693
+ pshufb m6, m52694
+ movu [r2 + r3], m62695
+2696
+ movu ym6, [r0 + mmsize]2697
+ vinserti32x8 m6, [r0 + r1 + mmsize], 12698
+ movu ym7, [r0 + mmsize + 8]2699
+ vinserti32x8 m7, [r0 + r1 + mmsize + 8], 12700
+2701
+ pshufb m8, m6, m32702
+ pshufb m6, m22703
+ pmaddwd m6, m02704
+ pmaddwd m8, m12705
+ paddd m6, m82706
+ paddd m6, m42707
+ psrad m6, INTERP_SHIFT_PS2708
+2709
+ pshufb m8, m7, m32710
+ pshufb m7, m22711
+ pmaddwd m7, m02712
+ pmaddwd m8, m12713
+ paddd m7, m82714
+ paddd m7, m42715
+ psrad m7, INTERP_SHIFT_PS2716
+2717
+ packssdw m6, m72718
+ pshufb m6, m52719
+ movu [r2 + mmsize], ym62720
+ vextracti32x8 [r2 + r3 + mmsize], m6, 12721
+%endmacro2722
+2723
+%macro PROCESS_IPFILTER_CHROMA_PS_48x1_AVX512 02724
+ ; register map2725
+ ; m0 , m1 - interpolate coeff2726
+ ; m2 , m3 - shuffle load order table2727
+ ; m4 - INTERP_OFFSET_PS2728
+ ; m5 - shuffle store order table2729
+2730
+ movu m6, [r0]2731
+ movu m7, [r0 + 8]2732
+2733
+ pshufb m8, m6, m32734
+ pshufb m6, m22735
+ pmaddwd m6, m02736
+ pmaddwd m8, m12737
+ paddd m6, m82738
+ paddd m6, m42739
+ psrad m6, INTERP_SHIFT_PS2740
+2741
+ pshufb m8, m7, m32742
+ pshufb m7, m22743
+ pmaddwd m7, m02744
+ pmaddwd m8, m12745
+ paddd m7, m82746
+ paddd m7, m42747
+ psrad m7, INTERP_SHIFT_PS2748
+2749
+ packssdw m6, m72750
+ pshufb m6, m52751
+ movu [r2], m62752
+2753
+ movu ym6, [r0 + mmsize]2754
+ movu ym7, [r0 + mmsize + 8]2755
+2756
+ pshufb ym8, ym6, ym32757
+ pshufb ym6, ym22758
+ pmaddwd ym6, ym02759
+ pmaddwd ym8, ym12760
+ paddd ym6, ym82761
+ paddd ym6, ym42762
+ psrad ym6, INTERP_SHIFT_PS2763
+2764
+ pshufb ym8, ym7, ym32765
+ pshufb ym7, ym22766
+ pmaddwd ym7, ym02767
+ pmaddwd ym8, ym12768
+ paddd ym7, ym82769
+ paddd ym7, ym42770
+ psrad ym7, INTERP_SHIFT_PS2771
+2772
+ packssdw ym6, ym72773
+ pshufb ym6, ym52774
+ movu [r2 + mmsize], ym62775
+%endmacro2776
+2777
+%if ARCH_X86_64 == 12778
+INIT_ZMM avx5122779
+cglobal interp_4tap_horiz_ps_48x64, 4,7,92780
+ shl r1d, 12781
+ shl r3d, 12782
+ mov r4d, r4m2783
+ mov r5d, r5m2784
+2785
+%ifdef PIC2786
+ lea r6, [tab_ChromaCoeff]2787
+ vpbroadcastd m0, [r6 + r4 * 8]2788
+ vpbroadcastd m1, [r6 + r4 * 8 + 4]2789
+%else2790
+ vpbroadcastd m0, [tab_ChromaCoeff + r4 * 8]2791
+ vpbroadcastd m1, [tab_ChromaCoeff + r4 * 8 + 4]2792
+%endif2793
+ vbroadcasti32x8 m2, [interp8_hpp_shuf1_load_avx512]2794
+ vbroadcasti32x8 m3, [interp8_hpp_shuf2_load_avx512]2795
+ vbroadcasti32x4 m4, [INTERP_OFFSET_PS]2796
+ vbroadcasti32x8 m5, [interp8_hpp_shuf1_store_avx512]2797
+2798
+ mov r6d, 642799
+ sub r0, 22800
+ test r5d, r5d2801
+ jz .loop2802
+ sub r0, r12803
+ add r6d, 32804
+ PROCESS_IPFILTER_CHROMA_PS_48x1_AVX5122805
+ lea r0, [r0 + r1]2806
+ lea r2, [r2 + r3]2807
+ dec r6d2808
+.loop:2809
+ PROCESS_IPFILTER_CHROMA_PS_48x2_AVX5122810
+ lea r0, [r0 + 2 * r1]2811
+ lea r2, [r2 + 2 * r3]2812
+ sub r6d, 22813
+ jnz .loop2814
+ RET2815
+%endif2816
+2817
+%macro PROCESS_IPFILTER_CHROMA_PS_8x4_AVX512 02818
+ ; register map2819
+ ; m0 , m1 - interpolate coeff2820
+ ; m2 , m3 - shuffle load order table2821
+ ; m4 - INTERP_OFFSET_PS2822
+ ; m5 - shuffle store order table2823
+2824
+ movu xm6, [r0]2825
+ vinserti32x4 m6, [r0 + r1], 12826
+ vinserti32x4 m6, [r0 + 2 * r1], 22827
+ vinserti32x4 m6, [r0 + r6], 32828
+2829
+ pshufb m8, m6, m32830
+ pshufb m6, m22831
+ pmaddwd m6, m02832
+ pmaddwd m8, m12833
+ paddd m6, m82834
+ paddd m6, m42835
+ psrad m6, INTERP_SHIFT_PS2836
+2837
+ movu xm7, [r0 + 8]2838
+ vinserti32x4 m7, [r0 + r1 + 8], 12839
+ vinserti32x4 m7, [r0 + 2 * r1 + 8], 22840
+ vinserti32x4 m7, [r0 + r6 + 8], 32841
+2842
+ pshufb m8, m7, m32843
+ pshufb m7, m22844
+ pmaddwd m7, m02845
+ pmaddwd m8, m12846
+ paddd m7, m82847
+ paddd m7, m42848
+ psrad m7, INTERP_SHIFT_PS2849
+2850
+ packssdw m6, m72851
+ pshufb m6, m52852
+ movu [r2], xm62853
+ vextracti32x4 [r2 + r3], m6, 12854
+ vextracti32x4 [r2 + 2 * r3], m6, 22855
+ vextracti32x4 [r2 + r7], m6, 32856
+%endmacro2857
+2858
+%macro PROCESS_IPFILTER_CHROMA_PS_8x3_AVX512 02859
+ movu xm6, [r0]2860
+ vinserti32x4 m6, [r0 + r1], 12861
+ vinserti32x4 m6, [r0 + 2 * r1], 22862
+2863
+ pshufb m8, m6, m32864
+ pshufb m6, m22865
+ pmaddwd m6, m02866
+ pmaddwd m8, m12867
+ paddd m6, m82868
+ paddd m6, m42869
+ psrad m6, INTERP_SHIFT_PS2870
+2871
+ movu xm7, [r0 + 8]2872
+ vinserti32x4 m7, [r0 + r1 + 8], 12873
+ vinserti32x4 m7, [r0 + 2 * r1 + 8], 22874
+2875
+ pshufb m8, m7, m32876
+ pshufb m7, m22877
+ pmaddwd m7, m02878
+ pmaddwd m8, m12879
+ paddd m7, m82880
+ paddd m7, m42881
+ psrad m7, INTERP_SHIFT_PS2882
+2883
+ packssdw m6, m72884
+ pshufb m6, m52885
+ movu [r2], xm62886
+ vextracti32x4 [r2 + r3], m6, 12887
+ vextracti32x4 [r2 + 2 * r3], m6, 22888
+%endmacro2889
+2890
+%macro IPFILTER_CHROMA_PS_AVX512_8xN 12891
+INIT_ZMM avx5122892
+cglobal interp_4tap_horiz_ps_8x%1, 4,9,92893
+ shl r1d, 12894
+ shl r3d, 12895
+ mov r4d, r4m2896
+ mov r5d, r5m2897
+2898
+ lea r6, [3 * r1]2899
+ lea r7, [3 * r3]2900
+%ifdef PIC2901
+ lea r8, [tab_ChromaCoeff]2902
+ vpbroadcastd m0, [r8 + r4 * 8]2903
+ vpbroadcastd m1, [r8 + r4 * 8 + 4]2904
+%else2905
+ vpbroadcastd m0, [tab_ChromaCoeff + r4 * 8]2906
+ vpbroadcastd m1, [tab_ChromaCoeff + r4 * 8 + 4]2907
+%endif2908
+ vbroadcasti32x8 m2, [interp8_hpp_shuf1_load_avx512]2909
+ vbroadcasti32x8 m3, [interp8_hpp_shuf2_load_avx512]2910
+ vbroadcasti32x4 m4, [INTERP_OFFSET_PS]2911
+ vbroadcasti32x8 m5, [interp8_hpp_shuf1_store_avx512]2912
+2913
+ mov r8d, %12914
+ sub r0, 22915
+ test r5d, r5d2916
+ jz .loop2917
+ sub r0, r12918
+ add r8d, 32919
+ PROCESS_IPFILTER_CHROMA_PS_8x3_AVX5122920
+ lea r0, [r0 + r6]2921
+ lea r2, [r2 + r7]2922
+ sub r8d, 32923
+2924
+.loop:2925
+ PROCESS_IPFILTER_CHROMA_PS_8x4_AVX5122926
+ lea r0, [r0 + 4 * r1]2927
+ lea r2, [r2 + 4 * r3]2928
+ sub r8d, 42929
+ jnz .loop2930
+ RET2931
+%endmacro2932
+2933
+%if ARCH_X86_642934
+IPFILTER_CHROMA_PS_AVX512_8xN 42935
+IPFILTER_CHROMA_PS_AVX512_8xN 82936
+IPFILTER_CHROMA_PS_AVX512_8xN 122937
+IPFILTER_CHROMA_PS_AVX512_8xN 162938
+IPFILTER_CHROMA_PS_AVX512_8xN 322939
+IPFILTER_CHROMA_PS_AVX512_8xN 642940
+%endif2941
+2942
+%macro PROCESS_IPFILTER_CHROMA_PS_24x4_AVX512 02943
+ ; register map2944
+ ; m0 , m1 - interpolate coeff2945
+ ; m2 , m3 - shuffle order table2946
+ ; m4 - INTERP_OFFSET_PS2947
+ ; m5 - shuffle store order table2948
+2949
+ movu ym6, [r0]2950
+ vinserti32x8 m6, [r0 + r1], 12951
+ movu ym7, [r0 + 8]2952
+ vinserti32x8 m7, [r0 + r1 + 8], 12953
+2954
+ pshufb m8, m6, m32955
+ pshufb m6, m22956
+ pmaddwd m6, m02957
+ pmaddwd m8, m12958
+ paddd m6, m82959
+ paddd m6, m42960
+ psrad m6, INTERP_SHIFT_PS2961
+2962
+ pshufb m8, m7, m32963
+ pshufb m7, m22964
+ pmaddwd m7, m02965
+ pmaddwd m8, m12966
+ paddd m7, m82967
+ paddd m7, m42968
+ psrad m7, INTERP_SHIFT_PS2969
+2970
+ packssdw m6, m72971
+ pshufb m6, m52972
+ movu [r2], ym62973
+ vextracti32x8 [r2 + r3], m6, 12974
+2975
+ movu ym6, [r0 + 2 * r1]2976
+ vinserti32x8 m6, [r0 + r6], 12977
+ movu ym7, [r0 + 2 * r1 + 8]2978
+ vinserti32x8 m7, [r0 + r6 + 8], 12979
+2980
+ pshufb m8, m6, m32981
+ pshufb m6, m22982
+ pmaddwd m6, m02983
+ pmaddwd m8, m12984
+ paddd m6, m82985
+ paddd m6, m42986
+ psrad m6, INTERP_SHIFT_PS2987
+2988
+ pshufb m8, m7, m32989
+ pshufb m7, m22990
+ pmaddwd m7, m02991
+ pmaddwd m8, m12992
+ paddd m7, m82993
+ paddd m7, m42994
+ psrad m7, INTERP_SHIFT_PS2995
+2996
+ packssdw m6, m72997
+ pshufb m6, m52998
+ movu [r2 + 2 * r3], ym62999
+ vextracti32x8 [r2 + r7], m6, 13000
+3001
+ movu xm6, [r0 + mmsize/2]3002
+ vinserti32x4 m6, [r0 + r1 + mmsize/2], 13003
+ vinserti32x4 m6, [r0 + 2 * r1 + mmsize/2], 23004
+ vinserti32x4 m6, [r0 + r6 + mmsize/2], 33005
+3006
+ pshufb m8, m6, m33007
+ pshufb m6, m23008
+ pmaddwd m6, m03009
+ pmaddwd m8, m13010
+ paddd m6, m83011
+ paddd m6, m43012
+ psrad m6, INTERP_SHIFT_PS3013
+3014
+ movu xm7, [r0 + mmsize/2 + 8]3015
+ vinserti32x4 m7, [r0 + r1 + mmsize/2 + 8], 13016
+ vinserti32x4 m7, [r0 + 2 * r1 + mmsize/2 + 8], 23017
+ vinserti32x4 m7, [r0 + r6 + mmsize/2 + 8], 33018
+3019
+ pshufb m8, m7, m33020
+ pshufb m7, m23021
+ pmaddwd m7, m03022
+ pmaddwd m8, m13023
+ paddd m7, m83024
+ paddd m7, m43025
+ psrad m7, INTERP_SHIFT_PS3026
+3027
+ packssdw m6, m73028
+ pshufb m6, m53029
+ movu [r2 + mmsize/2], xm63030
+ vextracti32x4 [r2 + r3 + mmsize/2], m6, 13031
+ vextracti32x4 [r2 + 2 * r3 + mmsize/2], m6, 23032
+ vextracti32x4 [r2 + r7 + mmsize/2], m6, 33033
+%endmacro3034
+3035
+%macro PROCESS_IPFILTER_CHROMA_PS_24x3_AVX512 03036
+ movu ym6, [r0]3037
+ vinserti32x8 m6, [r0 + r1], 13038
+ movu ym7, [r0 + 8]3039
+ vinserti32x8 m7, [r0 + r1 + 8], 13040
+3041
+ pshufb m8, m6, m33042
+ pshufb m6, m23043
+ pmaddwd m6, m03044
+ pmaddwd m8, m13045
+ paddd m6, m83046
+ paddd m6, m43047
+ psrad m6, INTERP_SHIFT_PS3048
+3049
+ pshufb m8, m7, m33050
+ pshufb m7, m23051
+ pmaddwd m7, m03052
+ pmaddwd m8, m13053
+ paddd m7, m83054
+ paddd m7, m43055
+ psrad m7, INTERP_SHIFT_PS3056
+3057
+ packssdw m6, m73058
+ pshufb m6, m53059
+ movu [r2], ym63060
+ vextracti32x8 [r2 + r3], m6, 13061
+3062
+ movu ym6, [r0 + 2 * r1]3063
+ movu ym7, [r0 + 2 * r1 + 8]3064
+3065
+ pshufb ym8, ym6, ym33066
+ pshufb ym6, ym23067
+ pmaddwd ym6, ym03068
+ pmaddwd ym8, ym13069
+ paddd ym6, ym83070
+ paddd ym6, ym43071
+ psrad ym6, INTERP_SHIFT_PS3072
+3073
+ pshufb ym8, ym7, ym33074
+ pshufb ym7, ym23075
+ pmaddwd ym7, ym03076
+ pmaddwd ym8, ym13077
+ paddd ym7, ym83078
+ paddd ym7, ym43079
+ psrad ym7, INTERP_SHIFT_PS3080
+3081
+ packssdw ym6, ym73082
+ pshufb ym6, ym53083
+ movu [r2 + 2 * r3], ym63084
+3085
+ movu xm6, [r0 + mmsize/2]3086
+ vinserti32x4 m6, [r0 + r1 + mmsize/2], 13087
+ vinserti32x4 m6, [r0 + 2 * r1 + mmsize/2], 23088
+3089
+ pshufb m8, m6, m33090
+ pshufb m6, m23091
+ pmaddwd m6, m03092
+ pmaddwd m8, m13093
+ paddd m6, m83094
+ paddd m6, m43095
+ psrad m6, INTERP_SHIFT_PS3096
+3097
+ movu xm7, [r0 + mmsize/2 + 8]3098
+ vinserti32x4 m7, [r0 + r1 + mmsize/2 + 8], 13099
+ vinserti32x4 m7, [r0 + 2 * r1 + mmsize/2 + 8], 23100
+3101
+ pshufb m8, m7, m33102
+ pshufb m7, m23103
+ pmaddwd m7, m03104
+ pmaddwd m8, m13105
+ paddd m7, m83106
+ paddd m7, m43107
+ psrad m7, INTERP_SHIFT_PS3108
+3109
+ packssdw m6, m73110
+ pshufb m6, m53111
+ movu [r2 + mmsize/2], xm63112
+ vextracti32x4 [r2 + r3 + mmsize/2], m6, 13113
+ vextracti32x4 [r2 + 2 * r3 + mmsize/2], m6, 23114
+%endmacro3115
+3116
+%macro IPFILTER_CHROMA_PS_AVX512_24xN 13117
+INIT_ZMM avx5123118
+cglobal interp_4tap_horiz_ps_24x%1, 4,9,93119
+ shl r1d, 13120
+ shl r3d, 13121
+ mov r4d, r4m3122
+ mov r5d, r5m3123
+3124
+ lea r6, [3 * r1]3125
+ lea r7, [3 * r3]3126
+%ifdef PIC3127
+ lea r8, [tab_ChromaCoeff]3128
+ vpbroadcastd m0, [r8 + r4 * 8]3129
+ vpbroadcastd m1, [r8 + r4 * 8 + 4]3130
+%else3131
+ vpbroadcastd m0, [tab_ChromaCoeff + r4 * 8]3132
+ vpbroadcastd m1, [tab_ChromaCoeff + r4 * 8 + 4]3133
+%endif3134
+ vbroadcasti32x8 m2, [interp8_hpp_shuf1_load_avx512]3135
+ vbroadcasti32x8 m3, [interp8_hpp_shuf2_load_avx512]3136
+ vbroadcasti32x4 m4, [INTERP_OFFSET_PS]3137
+ vbroadcasti32x8 m5,[interp8_hpp_shuf1_store_avx512]3138
+3139
+ mov r8d, %13140
+ sub r0, 23141
+ test r5d, r5d3142
+ jz .loop3143
+ sub r0, r13144
+ add r8d, 33145
+ PROCESS_IPFILTER_CHROMA_PS_24x3_AVX5123146
+ lea r0, [r0 + r6]3147
+ lea r2, [r2 + r7]3148
+ sub r8d, 33149
+3150
+.loop:3151
+ PROCESS_IPFILTER_CHROMA_PS_24x4_AVX5123152
+ lea r0, [r0 + 4 * r1]3153
+ lea r2, [r2 + 4 * r3]3154
+ sub r8d, 43155
+ jnz .loop3156
+ RET3157
+%endmacro3158
+3159
+%if ARCH_X86_643160
+IPFILTER_CHROMA_PS_AVX512_24xN 323161
+IPFILTER_CHROMA_PS_AVX512_24xN 643162
+%endif3163
+;-------------------------------------------------------------------------------------------------------------3164
+; avx512 chroma_hps code end3165
+;-------------------------------------------------------------------------------------------------------------3166
+;-------------------------------------------------------------------------------------------------------------3167
+; avx512 chroma_vps code start3168
+;-------------------------------------------------------------------------------------------------------------3169
+%macro PROCESS_CHROMA_VERT_PS_8x8_AVX512 03170
+ movu xm1, [r0]3171
+ lea r6, [r0 + 2 * r1]3172
+ lea r8, [r0 + 4 * r1]3173
+ lea r9, [r8 + 2 * r1]3174
+ vinserti32x4 m1, [r6], 13175
+ vinserti32x4 m1, [r8], 23176
+ vinserti32x4 m1, [r9], 33177
+ movu xm3, [r0 + r1]3178
+ vinserti32x4 m3, [r6 + r1], 13179
+ vinserti32x4 m3, [r8 + r1], 23180
+ vinserti32x4 m3, [r9 + r1], 33181
+ punpcklwd m0, m1, m33182
+ pmaddwd m0, [r5]3183
+ punpckhwd m1, m33184
+ pmaddwd m1, [r5]3185
+3186
+ movu xm4, [r0 + 2 * r1]3187
+ vinserti32x4 m4, [r6 + 2 * r1], 13188
+ vinserti32x4 m4, [r8 + 2 * r1], 23189
+ vinserti32x4 m4, [r9 + 2 * r1], 33190
+ punpcklwd m2, m3, m43191
+ pmaddwd m2, [r5]3192
+ punpckhwd m3, m43193
+ pmaddwd m3, [r5]3194
+3195
+ movu xm5, [r0 + r10]3196
+ vinserti32x4 m5, [r6 + r10], 13197
+ vinserti32x4 m5, [r8 + r10], 23198
+ vinserti32x4 m5, [r9 + r10], 33199
+ punpcklwd m6, m4, m53200
+ pmaddwd m6, [r5 + mmsize]3201
+ paddd m0, m63202
+ punpckhwd m4, m53203
+ pmaddwd m4, [r5 + mmsize]3204
+ paddd m1, m43205
+3206
+ movu xm4, [r0 + 4 * r1]3207
+ vinserti32x4 m4, [r6 + 4 * r1], 13208
+ vinserti32x4 m4, [r8 + 4 * r1], 23209
+ vinserti32x4 m4, [r9 + 4 * r1], 33210
+ punpcklwd m6, m5, m43211
+ pmaddwd m6, m93212
+ paddd m2, m63213
+ punpckhwd m5, m43214
+ pmaddwd m5, m93215
+ paddd m3, m53216
+3217
+ paddd m0, m73218
+ paddd m1, m73219
+ paddd m2, m73220
+ paddd m3, m73221
+3222
+ psrad m0, INTERP_SHIFT_PS3223
+ psrad m1, INTERP_SHIFT_PS3224
+ psrad m2, INTERP_SHIFT_PS3225
+ psrad m3, INTERP_SHIFT_PS3226
+3227
+ packssdw m0, m13228
+ packssdw m2, m33229
+ movu [r2], xm03230
+ movu [r2 + r3], xm23231
+ vextracti32x4 [r2 + 2 * r3], m0, 13232
+ vextracti32x4 [r2 + r7], m2, 13233
+ lea r2, [r2 + 4 * r3]3234
+ vextracti32x4 [r2], m0, 23235
+ vextracti32x4 [r2 + r3], m2, 23236
+ vextracti32x4 [r2 + 2 * r3], m0, 33237
+ vextracti32x4 [r2 + r7], m2, 33238
+%endmacro3239
+3240
+;-----------------------------------------------------------------------------------------------------------------3241
+; void interp_4tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)3242
+;-----------------------------------------------------------------------------------------------------------------3243
+%if ARCH_X86_643244
+INIT_ZMM avx5123245
+cglobal interp_4tap_vert_ps_8x8, 5, 11, 103246
+ add r1d, r1d3247
+ add r3d, r3d3248
+ sub r0, r13249
+ shl r4d, 73250
+3251
+%ifdef PIC3252
+ lea r5, [tab_ChromaCoeffV_avx512]3253
+ lea r5, [r5 + r4]3254
+%else3255
+ lea r5, [tab_ChromaCoeffV_avx512 + r4]3256
+%endif3257
+ vbroadcasti32x4 m7, [INTERP_OFFSET_PS]3258
+ lea r10, [3 * r1]3259
+ lea r7, [3 * r3]3260
+ mova m8, [r5]3261
+ mova m9, [r5 + mmsize]3262
+ PROCESS_CHROMA_VERT_PS_8x8_AVX5123263
+ RET3264
+%endif3265
+3266
+%macro FILTER_VER_PS_CHROMA_8xN_AVX512 13267
+INIT_ZMM avx5123268
+cglobal interp_4tap_vert_ps_8x%1, 5, 11, 103269
+ add r1d, r1d3270
+ add r3d, r3d3271
+ sub r0, r13272
+ shl r4d, 73273
+3274
+%ifdef PIC3275
+ lea r5, [tab_ChromaCoeffV_avx512]3276
+ lea r5, [r5 + r4]3277
+%else3278
+ lea r5, [tab_ChromaCoeffV_avx512 + r4]3279
+%endif3280
+ vbroadcasti32x4 m7, [INTERP_OFFSET_PS]3281
+ lea r10, [3 * r1]3282
+ lea r7, [3 * r3]3283
+ mova m8, [r5]3284
+ mova m9, [r5 + mmsize]3285
+%rep %1/8 - 13286
+ PROCESS_CHROMA_VERT_PS_8x8_AVX5123287
+ lea r0, [r8 + 4 * r1]3288
+ lea r2, [r2 + 4 * r3]3289
+%endrep3290
+ PROCESS_CHROMA_VERT_PS_8x8_AVX5123291
+ RET3292
+%endmacro3293
+3294
+%if ARCH_X86_643295
+FILTER_VER_PS_CHROMA_8xN_AVX512 163296
+FILTER_VER_PS_CHROMA_8xN_AVX512 323297
+FILTER_VER_PS_CHROMA_8xN_AVX512 643298
+%endif3299
+3300
+%macro PROCESS_CHROMA_VERT_PS_16x4_AVX512 03301
+ movu ym1, [r0]3302
+ lea r6, [r0 + 2 * r1]3303
+ vinserti32x8 m1, [r6], 13304
+ movu ym3, [r0 + r1]3305
+ vinserti32x8 m3, [r6 + r1], 13306
+ punpcklwd m0, m1, m33307
+ pmaddwd m0, m83308
+ punpckhwd m1, m33309
+ pmaddwd m1, m83310
+3311
+ movu ym4, [r0 + 2 * r1]3312
+ vinserti32x8 m4, [r6 + 2 * r1], 13313
+ punpcklwd m2, m3, m43314
+ pmaddwd m2, m83315
+ punpckhwd m3, m43316
+ pmaddwd m3, m83317
+3318
+ movu ym5, [r0 + r8]3319
+ vinserti32x8 m5, [r6 + r8], 13320
+ punpcklwd m6, m4, m53321
+ pmaddwd m6, m93322
+ paddd m0, m63323
+ punpckhwd m4, m53324
+ pmaddwd m4, m93325
+ paddd m1, m43326
+3327
+ movu ym4, [r0 + 4 * r1]3328
+ vinserti32x8 m4, [r6 + 4 * r1], 13329
+ punpcklwd m6, m5, m43330
+ pmaddwd m6, m93331
+ paddd m2, m63332
+ punpckhwd m5, m43333
+ pmaddwd m5, m93334
+ paddd m3, m53335
+3336
+ paddd m0, m73337
+ paddd m1, m73338
+ paddd m2, m73339
+ paddd m3, m73340
+3341
+ psrad m0, INTERP_SHIFT_PS3342
+ psrad m1, INTERP_SHIFT_PS3343
+ psrad m2, INTERP_SHIFT_PS3344
+ psrad m3, INTERP_SHIFT_PS3345
+3346
+ packssdw m0, m13347
+ packssdw m2, m33348
+ movu [r2], ym03349
+ movu [r2 + r3], ym23350
+ vextracti32x8 [r2 + 2 * r3], m0, 13351
+ vextracti32x8 [r2 + r7], m2, 13352
+%endmacro3353
+3354
+;-----------------------------------------------------------------------------------------------------------------3355
+; void interp_4tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)3356
+;-----------------------------------------------------------------------------------------------------------------3357
+%if ARCH_X86_643358
+INIT_ZMM avx5123359
+cglobal interp_4tap_vert_ps_16x4, 5, 9, 103360
+ add r1d, r1d3361
+ add r3d, r3d3362
+ sub r0, r13363
+ shl r4d, 73364
+3365
+%ifdef PIC3366
+ lea r5, [tab_ChromaCoeffV_avx512]3367
+ lea r5, [r5 + r4]3368
+%else3369
+ lea r5, [tab_ChromaCoeffV_avx512 + r4]3370
+%endif3371
+ vbroadcasti32x4 m7, [INTERP_OFFSET_PS]3372
+ lea r7, [3 * r3]3373
+ lea r8, [3 * r1]3374
+ mova m8, [r5]3375
+ mova m9, [r5 + mmsize]3376
+ PROCESS_CHROMA_VERT_PS_16x4_AVX5123377
+ RET3378
+%endif3379
+3380
+%macro FILTER_VER_PS_CHROMA_16xN_AVX512 13381
+INIT_ZMM avx5123382
+cglobal interp_4tap_vert_ps_16x%1, 5, 9, 103383
+ add r1d, r1d3384
+ add r3d, r3d3385
+ sub r0, r13386
+ shl r4d, 73387
+3388
+%ifdef PIC3389
+ lea r5, [tab_ChromaCoeffV_avx512]3390
+ lea r5, [r5 + r4]3391
+%else3392
+ lea r5, [tab_ChromaCoeffV_avx512 + r4]3393
+%endif3394
+ vbroadcasti32x4 m7, [INTERP_OFFSET_PS]3395
+ lea r7, [3 * r3]3396
+ lea r8, [3 * r1]3397
+ mova m8, [r5]3398
+ mova m9, [r5 + mmsize]3399
+%rep %1/4 - 13400
+ PROCESS_CHROMA_VERT_PS_16x4_AVX5123401
+ lea r0, [r0 + 4 * r1]3402
+ lea r2, [r2 + 4 * r3]3403
+%endrep3404
+ PROCESS_CHROMA_VERT_PS_16x4_AVX5123405
+ RET3406
+%endmacro3407
+3408
+%if ARCH_X86_643409
+FILTER_VER_PS_CHROMA_16xN_AVX512 83410
+FILTER_VER_PS_CHROMA_16xN_AVX512 123411
+FILTER_VER_PS_CHROMA_16xN_AVX512 163412
+FILTER_VER_PS_CHROMA_16xN_AVX512 243413
+FILTER_VER_PS_CHROMA_16xN_AVX512 323414
+FILTER_VER_PS_CHROMA_16xN_AVX512 643415
+%endif3416
+3417
+%macro PROCESS_CHROMA_VERT_PS_24x8_AVX512 03418
+ movu ym1, [r0]3419
+ lea r6, [r0 + 2 * r1]3420
+ lea r8, [r0 + 4 * r1]3421
+ lea r9, [r8 + 2 * r1]3422
+3423
+ movu ym10, [r8]3424
+ movu ym3, [r0 + r1]3425
+ movu ym12, [r8 + r1]3426
+ vinserti32x8 m1, [r6], 13427
+ vinserti32x8 m10, [r9], 13428
+ vinserti32x8 m3, [r6 + r1], 13429
+ vinserti32x8 m12, [r9 + r1], 13430
+3431
+ punpcklwd m0, m1, m33432
+ punpcklwd m9, m10, m123433
+ pmaddwd m0, m163434
+ pmaddwd m9, m163435
+ punpckhwd m1, m33436
+ punpckhwd m10, m123437
+ pmaddwd m1, m163438
+ pmaddwd m10, m163439
+3440
+ movu ym4, [r0 + 2 * r1]3441
+ movu ym13, [r8 + 2 * r1]3442
+ vinserti32x8 m4, [r6 + 2 * r1], 13443
+ vinserti32x8 m13, [r9 + 2 * r1], 13444
+ punpcklwd m2, m3, m43445
+ punpcklwd m11, m12, m133446
+ pmaddwd m2, m163447
+ pmaddwd m11, m163448
+ punpckhwd m3, m43449
+ punpckhwd m12, m133450
+ pmaddwd m3, m163451
+ pmaddwd m12, m163452
+3453
+ movu ym5, [r0 + r10]3454
+ vinserti32x8 m5, [r6 + r10], 13455
+ movu ym14, [r8 + r10]3456
+ vinserti32x8 m14, [r9 + r10], 13457
+ punpcklwd m6, m4, m53458
+ punpcklwd m15, m13, m143459
+ pmaddwd m6, m173460
+ pmaddwd m15, m173461
+ paddd m0, m63462
+ paddd m9, m153463
+ punpckhwd m4, m53464
+ punpckhwd m13, m143465
+ pmaddwd m4, m173466
+ pmaddwd m13, m173467
+ paddd m1, m43468
+ paddd m10, m133469
+3470
+ movu ym4, [r0 + 4 * r1]3471
+ vinserti32x8 m4, [r6 + 4 * r1], 13472
+ movu ym13, [r8 + 4 * r1]3473
+ vinserti32x8 m13, [r9 + 4 * r1], 13474
+ punpcklwd m6, m5, m43475
+ punpcklwd m15, m14, m133476
+ pmaddwd m6, m173477
+ pmaddwd m15, m173478
+ paddd m2, m63479
+ paddd m11, m153480
+ punpckhwd m5, m43481
+ punpckhwd m14, m133482
+ pmaddwd m5, m173483
+ pmaddwd m14, m173484
+ paddd m3, m53485
+ paddd m12, m143486
+3487
+ paddd m0, m73488
+ paddd m1, m73489
+ paddd m2, m73490
+ paddd m3, m73491
+ paddd m9, m73492
+ paddd m10, m73493
+ paddd m11, m73494
+ paddd m12, m73495
+3496
+ psrad m0, INTERP_SHIFT_PS3497
+ psrad m1, INTERP_SHIFT_PS3498
+ psrad m2, INTERP_SHIFT_PS3499
+ psrad m3, INTERP_SHIFT_PS3500
+ psrad m9, INTERP_SHIFT_PS3501
+ psrad m10, INTERP_SHIFT_PS3502
+ psrad m11, INTERP_SHIFT_PS3503
+ psrad m12, INTERP_SHIFT_PS3504
+3505
+ packssdw m0, m13506
+ packssdw m2, m33507
+ packssdw m9, m103508
+ packssdw m11, m123509
+ movu [r2], ym03510
+ movu [r2 + r3], ym23511
+ vextracti32x8 [r2 + 2 * r3], m0, 13512
+ vextracti32x8 [r2 + r7], m2, 13513
+ lea r11, [r2 + 4 * r3]3514
+ movu [r11], ym93515
+ movu [r11 + r3], ym113516
+ vextracti32x8 [r11 + 2 * r3], m9, 13517
+ vextracti32x8 [r11 + r7], m11, 13518
+3519
+ movu xm1, [r0 + mmsize/2]3520
+ vinserti32x4 m1, [r6 + mmsize/2], 13521
+ vinserti32x4 m1, [r8 + mmsize/2], 23522
+ vinserti32x4 m1, [r9 + mmsize/2], 33523
+ movu xm3, [r0 + r1 + mmsize/2]3524
+ vinserti32x4 m3, [r6 + r1 + mmsize/2], 13525
+ vinserti32x4 m3, [r8 + r1 + mmsize/2], 23526
+ vinserti32x4 m3, [r9 + r1 + mmsize/2], 33527
+ punpcklwd m0, m1, m33528
+ pmaddwd m0, m163529
+ punpckhwd m1, m33530
+ pmaddwd m1, m163531
+3532
+ movu xm4, [r0 + 2 * r1 + mmsize/2]3533
+ vinserti32x4 m4, [r6 + 2 * r1 + mmsize/2], 13534
+ vinserti32x4 m4, [r8 + 2 * r1 + mmsize/2], 23535
+ vinserti32x4 m4, [r9 + 2 * r1 + mmsize/2], 33536
+ punpcklwd m2, m3, m43537
+ pmaddwd m2, m163538
+ punpckhwd m3, m43539
+ pmaddwd m3, m163540
+3541
+ movu xm5, [r0 + r10 + mmsize/2]3542
+ vinserti32x4 m5, [r6 + r10 + mmsize/2], 13543
+ vinserti32x4 m5, [r8 + r10 + mmsize/2], 23544
+ vinserti32x4 m5, [r9 + r10 + mmsize/2], 33545
+ punpcklwd m6, m4, m53546
+ pmaddwd m6, m173547
+ paddd m0, m63548
+ punpckhwd m4, m53549
+ pmaddwd m4, m173550
+ paddd m1, m43551
+3552
+ movu xm4, [r0 + 4 * r1 + mmsize/2]3553
+ vinserti32x4 m4, [r6 + 4 * r1 + mmsize/2], 13554
+ vinserti32x4 m4, [r8 + 4 * r1 + mmsize/2], 23555
+ vinserti32x4 m4, [r9 + 4 * r1 + mmsize/2], 33556
+ punpcklwd m6, m5, m43557
+ pmaddwd m6, m173558
+ paddd m2, m63559
+ punpckhwd m5, m43560
+ pmaddwd m5, m173561
+ paddd m3, m53562
+3563
+ paddd m0, m73564
+ paddd m1, m73565
+ paddd m2, m73566
+ paddd m3, m73567
+3568
+ psrad m0, INTERP_SHIFT_PS3569
+ psrad m1, INTERP_SHIFT_PS3570
+ psrad m2, INTERP_SHIFT_PS3571
+ psrad m3, INTERP_SHIFT_PS3572
+3573
+ packssdw m0, m13574
+ packssdw m2, m33575
+ movu [r2 + mmsize/2], xm03576
+ movu [r2 + r3 + mmsize/2], xm23577
+ vextracti32x4 [r2 + 2 * r3 + mmsize/2], m0, 13578
+ vextracti32x4 [r2 + r7 + mmsize/2], m2, 13579
+ lea r2, [r2 + 4 * r3]3580
+ vextracti32x4 [r2 + mmsize/2], m0, 23581
+ vextracti32x4 [r2 + r3 + mmsize/2], m2, 23582
+ vextracti32x4 [r2 + 2 * r3 + mmsize/2], m0, 33583
+ vextracti32x4 [r2 + r7 + mmsize/2], m2, 33584
+%endmacro3585
+3586
+%macro FILTER_VER_PS_CHROMA_24xN_AVX512 13587
+INIT_ZMM avx5123588
+cglobal interp_4tap_vert_ps_24x%1, 5, 12, 183589
+ add r1d, r1d3590
+ add r3d, r3d3591
+ sub r0, r13592
+ shl r4d, 73593
+3594
+%ifdef PIC3595
+ lea r5, [tab_ChromaCoeffV_avx512]3596
+ lea r5, [r5 + r4]3597
+%else3598
+ lea r5, [tab_ChromaCoeffV_avx512 + r4]3599
+%endif3600
+ vbroadcasti32x4 m7, [INTERP_OFFSET_PS]3601
+ lea r10, [3 * r1]3602
+ lea r7, [3 * r3]3603
+ mova m16, [r5]3604
+ mova m17, [r5 + mmsize]3605
+%rep %1/8 - 13606
+ PROCESS_CHROMA_VERT_PS_24x8_AVX5123607
+ lea r0, [r8 + 4 * r1]3608
+ lea r2, [r2 + 4 * r3]3609
+%endrep3610
+ PROCESS_CHROMA_VERT_PS_24x8_AVX5123611
+ RET3612
+%endmacro3613
+3614
+%if ARCH_X86_643615
+ FILTER_VER_PS_CHROMA_24xN_AVX512 323616
+ FILTER_VER_PS_CHROMA_24xN_AVX512 643617
+%endif3618
+3619
+%macro PROCESS_CHROMA_VERT_PS_32x2_AVX512 03620
+ movu m1, [r0]3621
+ movu m3, [r0 + r1]3622
+ punpcklwd m0, m1, m33623
+ pmaddwd m0, m93624
+ punpckhwd m1, m33625
+ pmaddwd m1, m93626
+3627
+ movu m4, [r0 + 2 * r1]3628
+ punpcklwd m2, m3, m43629
+ pmaddwd m2, m93630
+ punpckhwd m3, m43631
+ pmaddwd m3, m93632
+3633
+ lea r0, [r0 + 2 * r1]3634
+ movu m5, [r0 + r1]3635
+ punpcklwd m6, m4, m53636
+ pmaddwd m6, m103637
+ paddd m0, m63638
+ punpckhwd m4, m53639
+ pmaddwd m4, m103640
+ paddd m1, m43641
+3642
+ movu m4, [r0 + 2 * r1]3643
+ punpcklwd m6, m5, m43644
+ pmaddwd m6, m103645
+ paddd m2, m63646
+ punpckhwd m5, m43647
+ pmaddwd m5, m103648
+ paddd m3, m53649
+3650
+ paddd m0, m73651
+ paddd m1, m73652
+ paddd m2, m73653
+ paddd m3, m73654
+ psrad m0, INTERP_SHIFT_PS3655
+ psrad m1, INTERP_SHIFT_PS3656
+ psrad m2, INTERP_SHIFT_PS3657
+ psrad m3, INTERP_SHIFT_PS3658
+3659
+ packssdw m0, m13660
+ packssdw m2, m33661
+ movu [r2], m03662
+ movu [r2 + r3], m23663
+%endmacro3664
+3665
+;-----------------------------------------------------------------------------------------------------------------3666
+; void interp_4tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)3667
+;-----------------------------------------------------------------------------------------------------------------3668
+%macro FILTER_VER_PS_CHROMA_32xN_AVX512 13669
+INIT_ZMM avx5123670
+cglobal interp_4tap_vert_ps_32x%1, 5, 7, 113671
+ add r1d, r1d3672
+ add r3d, r3d3673
+ sub r0, r13674
+ shl r4d, 73675
+3676
+%ifdef PIC3677
+ lea r5, [tab_ChromaCoeffV_avx512]3678
+ lea r5, [r5 + r4]3679
+%else3680
+ lea r5, [tab_ChromaCoeffV_avx512 + r4]3681
+%endif3682
+ vbroadcasti32x4 m7, [INTERP_OFFSET_PS]3683
+ mova m9, [r5]3684
+ mova m10, [r5 + mmsize]3685
+%rep %1/2 - 13686
+ PROCESS_CHROMA_VERT_PS_32x2_AVX5123687
+ lea r2, [r2 + 2 * r3]3688
+%endrep3689
+ PROCESS_CHROMA_VERT_PS_32x2_AVX5123690
+ RET3691
+%endmacro3692
+3693
+%if ARCH_X86_643694
+FILTER_VER_PS_CHROMA_32xN_AVX512 83695
+FILTER_VER_PS_CHROMA_32xN_AVX512 163696
+FILTER_VER_PS_CHROMA_32xN_AVX512 243697
+FILTER_VER_PS_CHROMA_32xN_AVX512 323698
+FILTER_VER_PS_CHROMA_32xN_AVX512 483699
+FILTER_VER_PS_CHROMA_32xN_AVX512 643700
+%endif3701
+3702
+%macro PROCESS_CHROMA_VERT_PS_48x4_AVX512 03703
+ movu m1, [r0]3704
+ lea r6, [r0 + 2 * r1]3705
+ movu m10, [r6]3706
+ movu m3, [r0 + r1]3707
+ movu m12, [r6 + r1]3708
+ punpcklwd m0, m1, m33709
+ punpcklwd m9, m10, m123710
+ pmaddwd m0, m163711
+ pmaddwd m9, m163712
+ punpckhwd m1, m33713
+ punpckhwd m10, m123714
+ pmaddwd m1, m163715
+ pmaddwd m10, m163716
+3717
+ movu m4, [r0 + 2 * r1]3718
+ movu m13, [r6 + 2 * r1]3719
+ punpcklwd m2, m3, m43720
+ punpcklwd m11, m12, m133721
+ pmaddwd m2, m163722
+ pmaddwd m11, m163723
+ punpckhwd m3, m43724
+ punpckhwd m12, m133725
+ pmaddwd m3, m163726
+ pmaddwd m12, m163727
+3728
+ movu m5, [r0 + r7]3729
+ movu m14, [r6 + r7]3730
+ punpcklwd m6, m4, m53731
+ punpcklwd m15, m13, m143732
+ pmaddwd m6, m173733
+ pmaddwd m15, m173734
+ paddd m0, m63735
+ paddd m9, m153736
+ punpckhwd m4, m53737
+ punpckhwd m13, m143738
+ pmaddwd m4, m173739
+ pmaddwd m13, m173740
+ paddd m1, m43741
+ paddd m10, m133742
+3743
+ movu m4, [r0 + 4 * r1]3744
+ movu m13, [r6 + 4 * r1]3745
+ punpcklwd m6, m5, m43746
+ punpcklwd m15, m14, m133747
+ pmaddwd m6, m173748
+ pmaddwd m15, m173749
+ paddd m2, m63750
+ paddd m11, m153751
+ punpckhwd m5, m43752
+ punpckhwd m14, m133753
+ pmaddwd m5, m173754
+ pmaddwd m14, m173755
+ paddd m3, m53756
+ paddd m12, m143757
+3758
+ paddd m0, m73759
+ paddd m1, m73760
+ paddd m2, m73761
+ paddd m3, m73762
+ paddd m9, m73763
+ paddd m10, m73764
+ paddd m11, m73765
+ paddd m12, m73766
+3767
+ psrad m0, INTERP_SHIFT_PS3768
+ psrad m1, INTERP_SHIFT_PS3769
+ psrad m2, INTERP_SHIFT_PS3770
+ psrad m3, INTERP_SHIFT_PS3771
+ psrad m9, INTERP_SHIFT_PS3772
+ psrad m10, INTERP_SHIFT_PS3773
+ psrad m11, INTERP_SHIFT_PS3774
+ psrad m12, INTERP_SHIFT_PS3775
+3776
+ packssdw m0, m13777
+ packssdw m2, m33778
+ packssdw m9, m103779
+ packssdw m11, m123780
+ movu [r2], m03781
+ movu [r2 + r3], m23782
+ movu [r2 + 2 * r3], m93783
+ movu [r2 + r8], m113784
+3785
+ movu ym1, [r0 + mmsize]3786
+ vinserti32x8 m1, [r6 + mmsize], 13787
+ movu ym3, [r0 + r1 + mmsize]3788
+ vinserti32x8 m3, [r6 + r1 + mmsize], 13789
+ punpcklwd m0, m1, m33790
+ pmaddwd m0, m163791
+ punpckhwd m1, m33792
+ pmaddwd m1, m163793
+3794
+ movu ym4, [r0 + 2 * r1 + mmsize]3795
+ vinserti32x8 m4, [r6 + 2 * r1 + mmsize], 13796
+ punpcklwd m2, m3, m43797
+ pmaddwd m2, m163798
+ punpckhwd m3, m43799
+ pmaddwd m3, m163800
+3801
+ movu ym5, [r0 + r7 + mmsize]3802
+ vinserti32x8 m5, [r6 + r7 + mmsize], 13803
+ punpcklwd m6, m4, m53804
+ pmaddwd m6, m173805
+ paddd m0, m63806
+ punpckhwd m4, m53807
+ pmaddwd m4, m173808
+ paddd m1, m43809
+3810
+ movu ym4, [r0 + 4 * r1 + mmsize]3811
+ vinserti32x8 m4, [r6 + 4 * r1 + mmsize], 13812
+ punpcklwd m6, m5, m43813
+ pmaddwd m6, m173814
+ paddd m2, m63815
+ punpckhwd m5, m43816
+ pmaddwd m5, m173817
+ paddd m3, m53818
+3819
+ paddd m0, m73820
+ paddd m1, m73821
+ paddd m2, m73822
+ paddd m3, m73823
+3824
+ psrad m0, INTERP_SHIFT_PS3825
+ psrad m1, INTERP_SHIFT_PS3826
+ psrad m2, INTERP_SHIFT_PS3827
+ psrad m3, INTERP_SHIFT_PS3828
+3829
+ packssdw m0, m13830
+ packssdw m2, m33831
+ movu [r2 + mmsize], ym03832
+ movu [r2 + r3 + mmsize], ym23833
+ vextracti32x8 [r2 + 2 * r3 + mmsize], m0, 13834
+ vextracti32x8 [r2 + r8 + mmsize], m2, 13835
+%endmacro3836
+3837
+%if ARCH_X86_643838
+INIT_ZMM avx5123839
+cglobal interp_4tap_vert_ps_48x64, 5, 9, 183840
+ add r1d, r1d3841
+ add r3d, r3d3842
+ sub r0, r13843
+ shl r4d, 73844
+%ifdef PIC3845
+ lea r5, [tab_ChromaCoeffV_avx512]3846
+ lea r5, [r5 + r4]3847
+%else3848
+ lea r5, [tab_ChromaCoeffV_avx512 + r4]3849
+%endif3850
+ lea r7, [3 * r1]3851
+ lea r8, [3 * r3]3852
+ vbroadcasti32x4 m7, [INTERP_OFFSET_PS]3853
+ mova m16, [r5]3854
+ mova m17, [r5 + mmsize]3855
+%rep 153856
+ PROCESS_CHROMA_VERT_PS_48x4_AVX5123857
+ lea r0, [r0 + 4 * r1]3858
+ lea r2, [r2 + 4 * r3]3859
+%endrep3860
+ PROCESS_CHROMA_VERT_PS_48x4_AVX5123861
+ RET3862
+%endif3863
+3864
+%macro PROCESS_CHROMA_VERT_PS_64x2_AVX512 03865
+ movu m1, [r0]3866
+ movu m3, [r0 + r1]3867
+ punpcklwd m0, m1, m33868
+ pmaddwd m0, m153869
+ punpckhwd m1, m33870
+ pmaddwd m1, m153871
+3872
+ movu m9, [r0 + mmsize]3873
+ movu m11, [r0 + r1 + mmsize]3874
+ punpcklwd m8, m9, m113875
+ pmaddwd m8, m153876
+ punpckhwd m9, m113877
+ pmaddwd m9, m153878
+3879
+ movu m4, [r0 + 2 * r1]3880
+ punpcklwd m2, m3, m43881
+ pmaddwd m2, m153882
+ punpckhwd m3, m43883
+ pmaddwd m3, m153884
+3885
+ movu m12, [r0 + 2 * r1 + mmsize]3886
+ punpcklwd m10, m11, m123887
+ pmaddwd m10, m153888
+ punpckhwd m11, m123889
+ pmaddwd m11, m153890
+3891
+ lea r0, [r0 + 2 * r1]3892
+ movu m5, [r0 + r1]3893
+ punpcklwd m6, m4, m53894
+ pmaddwd m6, m163895
+ paddd m0, m63896
+ punpckhwd m4, m53897
+ pmaddwd m4, m163898
+ paddd m1, m43899
+3900
+ movu m13, [r0 + r1 + mmsize]3901
+ punpcklwd m14, m12, m133902
+ pmaddwd m14, m163903
+ paddd m8, m143904
+ punpckhwd m12, m133905
+ pmaddwd m12, m163906
+ paddd m9, m123907
+3908
+ movu m4, [r0 + 2 * r1]3909
+ punpcklwd m6, m5, m43910
+ pmaddwd m6, m163911
+ paddd m2, m63912
+ punpckhwd m5, m43913
+ pmaddwd m5, m163914
+ paddd m3, m53915
+3916
+ movu m12, [r0 + 2 * r1 + mmsize]3917
+ punpcklwd m14, m13, m123918
+ pmaddwd m14, m163919
+ paddd m10, m143920
+ punpckhwd m13, m123921
+ pmaddwd m13, m163922
+ paddd m11, m133923
+3924
+ paddd m0, m73925
+ paddd m1, m73926
+ paddd m2, m73927
+ paddd m3, m73928
+ paddd m8, m73929
+ paddd m9, m73930
+ paddd m10, m73931
+ paddd m11, m73932
+3933
+ psrad m0, INTERP_SHIFT_PS3934
+ psrad m1, INTERP_SHIFT_PS3935
+ psrad m2, INTERP_SHIFT_PS3936
+ psrad m3, INTERP_SHIFT_PS3937
+ psrad m8, INTERP_SHIFT_PS3938
+ psrad m9, INTERP_SHIFT_PS3939
+ psrad m10, INTERP_SHIFT_PS3940
+ psrad m11, INTERP_SHIFT_PS3941
+3942
+ packssdw m0, m13943
+ packssdw m2, m33944
+ packssdw m8, m93945
+ packssdw m10, m113946
+ movu [r2], m03947
+ movu [r2 + r3], m23948
+ movu [r2 + mmsize], m83949
+ movu [r2 + r3 + mmsize], m103950
+%endmacro3951
+3952
+;-----------------------------------------------------------------------------------------------------------------3953
+; void interp_4tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)3954
+;-----------------------------------------------------------------------------------------------------------------3955
+%macro FILTER_VER_PS_CHROMA_64xN_AVX512 13956
+INIT_ZMM avx5123957
+cglobal interp_4tap_vert_ps_64x%1, 5, 7, 173958
+ add r1d, r1d3959
+ add r3d, r3d3960
+ sub r0, r13961
+ shl r4d, 73962
+3963
+%ifdef PIC3964
+ lea r5, [tab_ChromaCoeffV_avx512]3965
+ lea r5, [r5 + r4]3966
+%else3967
+ lea r5, [tab_ChromaCoeffV_avx512 + r4]3968
+%endif3969
+ vbroadcasti32x4 m7, [INTERP_OFFSET_PS]3970
+ mova m15, [r5]3971
+ mova m16, [r5 + mmsize]3972
+3973
+%rep %1/2 - 13974
+ PROCESS_CHROMA_VERT_PS_64x2_AVX5123975
+ lea r2, [r2 + 2 * r3]3976
+%endrep3977
+ PROCESS_CHROMA_VERT_PS_64x2_AVX5123978
+ RET3979
+%endmacro3980
+3981
+%if ARCH_X86_643982
+FILTER_VER_PS_CHROMA_64xN_AVX512 163983
+FILTER_VER_PS_CHROMA_64xN_AVX512 323984
+FILTER_VER_PS_CHROMA_64xN_AVX512 483985
+FILTER_VER_PS_CHROMA_64xN_AVX512 643986
+%endif3987
+;-------------------------------------------------------------------------------------------------------------3988
+; avx512 chroma_vps code end3989
+;-------------------------------------------------------------------------------------------------------------3990
+;-------------------------------------------------------------------------------------------------------------3991
+; avx512 chroma_vsp and chroma_vss code start3992
+;-------------------------------------------------------------------------------------------------------------3993
+%macro PROCESS_CHROMA_VERT_S_8x8_AVX512 13994
+ movu xm1, [r0]3995
+ lea r6, [r0 + 2 * r1]3996
+ lea r8, [r0 + 4 * r1]3997
+ lea r9, [r8 + 2 * r1]3998
+ vinserti32x4 m1, [r6], 13999
+ vinserti32x4 m1, [r8], 24000
+ vinserti32x4 m1, [r9], 34001
+ movu xm3, [r0 + r1]4002
+ vinserti32x4 m3, [r6 + r1], 14003
+ vinserti32x4 m3, [r8 + r1], 24004
+ vinserti32x4 m3, [r9 + r1], 34005
+ punpcklwd m0, m1, m34006
+ pmaddwd m0, m84007
+ punpckhwd m1, m34008
+ pmaddwd m1, m84009
+4010
+ movu xm4, [r0 + 2 * r1]4011
+ vinserti32x4 m4, [r6 + 2 * r1], 14012
+ vinserti32x4 m4, [r8 + 2 * r1], 24013
+ vinserti32x4 m4, [r9 + 2 * r1], 34014
+ punpcklwd m2, m3, m44015
+ pmaddwd m2, m84016
+ punpckhwd m3, m44017
+ pmaddwd m3, m84018
+4019
+ movu xm5, [r0 + r10]4020
+ vinserti32x4 m5, [r6 + r10], 14021
+ vinserti32x4 m5, [r8 + r10], 24022
+ vinserti32x4 m5, [r9 + r10], 34023
+ punpcklwd m6, m4, m54024
+ pmaddwd m6, m94025
+ paddd m0, m64026
+ punpckhwd m4, m54027
+ pmaddwd m4, m94028
+ paddd m1, m44029
+4030
+ movu xm4, [r0 + 4 * r1]4031
+ vinserti32x4 m4, [r6 + 4 * r1], 14032
+ vinserti32x4 m4, [r8 + 4 * r1], 24033
+ vinserti32x4 m4, [r9 + 4 * r1], 34034
+ punpcklwd m6, m5, m44035
+ pmaddwd m6, m94036
+ paddd m2, m64037
+ punpckhwd m5, m44038
+ pmaddwd m5, m94039
+ paddd m3, m54040
+4041
+%ifidn %1,sp4042
+ paddd m0, m74043
+ paddd m1, m74044
+ paddd m2, m74045
+ paddd m3, m74046
+4047
+ psrad m0, INTERP_SHIFT_SP4048
+ psrad m1, INTERP_SHIFT_SP4049
+ psrad m2, INTERP_SHIFT_SP4050
+ psrad m3, INTERP_SHIFT_SP4051
+4052
+ packssdw m0, m14053
+ packssdw m2, m34054
+ CLIPW2 m0, m2, m10, m114055
+%else4056
+ psrad m0, 64057
+ psrad m1, 64058
+ psrad m2, 64059
+ psrad m3, 64060
+ packssdw m0, m14061
+ packssdw m2, m34062
+%endif4063
+4064
+ movu [r2], xm04065
+ movu [r2 + r3], xm24066
+ vextracti32x4 [r2 + 2 * r3], m0, 14067
+ vextracti32x4 [r2 + r7], m2, 14068
+ lea r2, [r2 + 4 * r3]4069
+ vextracti32x4 [r2], m0, 24070
+ vextracti32x4 [r2 + r3], m2, 24071
+ vextracti32x4 [r2 + 2 * r3], m0, 34072
+ vextracti32x4 [r2 + r7], m2, 34073
+%endmacro4074
+4075
+;-----------------------------------------------------------------------------------------------------------------4076
+; void interp_4tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)4077
+;-----------------------------------------------------------------------------------------------------------------4078
+%macro CHROMA_VERT_S_8x8_AVX512 14079
+INIT_ZMM avx5124080
+cglobal interp_4tap_vert_%1_8x8, 5, 11, 124081
+ add r1d, r1d4082
+ add r3d, r3d4083
+ sub r0, r14084
+ shl r4d, 74085
+%ifdef PIC4086
+ lea r5, [tab_ChromaCoeffV_avx512]4087
+ mova m8, [r5 + r4]4088
+ mova m9, [r5 + r4 + mmsize]4089
+%else4090
+ lea r5, [tab_ChromaCoeffV_avx512 + r4]4091
+ mova m8, [r5]4092
+ mova m9, [r5 + mmsize]4093
+%endif4094
+%ifidn %1, sp4095
+ vbroadcasti32x4 m7, [INTERP_OFFSET_SP]4096
+ pxor m10, m104097
+ vbroadcasti32x8 m11, [pw_pixel_max]4098
+%endif4099
+ lea r10, [3 * r1]4100
+ lea r7, [3 * r3]4101
+4102
+ PROCESS_CHROMA_VERT_S_8x8_AVX512 %14103
+ RET4104
+%endmacro4105
+4106
+%if ARCH_X86_644107
+ CHROMA_VERT_S_8x8_AVX512 ss4108
+ CHROMA_VERT_S_8x8_AVX512 sp4109
+%endif4110
+%macro FILTER_VER_S_CHROMA_8xN_AVX512 24111
+INIT_ZMM avx5124112
+cglobal interp_4tap_vert_%1_8x%2, 5, 11, 104113
+ add r1d, r1d4114
+ add r3d, r3d4115
+ sub r0, r14116
+ shl r4d, 74117
+%ifdef PIC4118
+ lea r5, [tab_ChromaCoeffV_avx512]4119
+ mova m8, [r5 + r4]4120
+ mova m9, [r5 + r4 + mmsize]4121
+%else4122
+ lea r5, [tab_ChromaCoeffV_avx512 + r4]4123
+ mova m8, [r5]4124
+ mova m9, [r5 + mmsize]4125
+%endif4126
+4127
+%ifidn %1, sp4128
+ vbroadcasti32x4 m7, [INTERP_OFFSET_SP]4129
+ pxor m10, m104130
+ vbroadcasti32x8 m11, [pw_pixel_max]4131
+%endif4132
+ lea r10, [3 * r1]4133
+ lea r7, [3 * r3]4134
+4135
+%rep %2/8 - 14136
+ PROCESS_CHROMA_VERT_S_8x8_AVX512 %14137
+ lea r0, [r8 + 4 * r1]4138
+ lea r2, [r2 + 4 * r3]4139
+%endrep4140
+ PROCESS_CHROMA_VERT_S_8x8_AVX512 %14141
+ RET4142
+%endmacro4143
+%if ARCH_X86_644144
+ FILTER_VER_S_CHROMA_8xN_AVX512 ss, 164145
+ FILTER_VER_S_CHROMA_8xN_AVX512 ss, 324146
+ FILTER_VER_S_CHROMA_8xN_AVX512 ss, 644147
+ FILTER_VER_S_CHROMA_8xN_AVX512 sp, 164148
+ FILTER_VER_S_CHROMA_8xN_AVX512 sp, 324149
+ FILTER_VER_S_CHROMA_8xN_AVX512 sp, 644150
+%endif4151
+%macro PROCESS_CHROMA_VERT_S_16x4_AVX512 14152
+ movu ym1, [r0]4153
+ lea r6, [r0 + 2 * r1]4154
+ vinserti32x8 m1, [r6], 14155
+ movu ym3, [r0 + r1]4156
+ vinserti32x8 m3, [r6 + r1], 14157
+ punpcklwd m0, m1, m34158
+ pmaddwd m0, m84159
+ punpckhwd m1, m34160
+ pmaddwd m1, m84161
+4162
+ movu ym4, [r0 + 2 * r1]4163
+ vinserti32x8 m4, [r6 + 2 * r1], 14164
+ punpcklwd m2, m3, m44165
+ pmaddwd m2, m84166
+ punpckhwd m3, m44167
+ pmaddwd m3, m84168
+4169
+ movu ym5, [r0 + r8]4170
+ vinserti32x8 m5, [r6 + r8], 14171
+ punpcklwd m6, m4, m54172
+ pmaddwd m6, m94173
+ paddd m0, m64174
+ punpckhwd m4, m54175
+ pmaddwd m4, m94176
+ paddd m1, m44177
+4178
+ movu ym4, [r0 + 4 * r1]4179
+ vinserti32x8 m4, [r6 + 4 * r1], 14180
+ punpcklwd m6, m5, m44181
+ pmaddwd m6, m94182
+ paddd m2, m64183
+ punpckhwd m5, m44184
+ pmaddwd m5, m94185
+ paddd m3, m54186
+4187
+%ifidn %1,sp4188
+ paddd m0, m74189
+ paddd m1, m74190
+ paddd m2, m74191
+ paddd m3, m74192
+4193
+ psrad m0, INTERP_SHIFT_SP4194
+ psrad m1, INTERP_SHIFT_SP4195
+ psrad m2, INTERP_SHIFT_SP4196
+ psrad m3, INTERP_SHIFT_SP4197
+4198
+ packssdw m0, m14199
+ packssdw m2, m34200
+ CLIPW2 m0, m2, m10, m114201
+%else4202
+ psrad m0, 64203
+ psrad m1, 64204
+ psrad m2, 64205
+ psrad m3, 64206
+ packssdw m0, m14207
+ packssdw m2, m34208
+%endif4209
+4210
+ movu [r2], ym04211
+ movu [r2 + r3], ym24212
+ vextracti32x8 [r2 + 2 * r3], m0, 14213
+ vextracti32x8 [r2 + r7], m2, 14214
+%endmacro4215
+4216
+;-----------------------------------------------------------------------------------------------------------------4217
+; void interp_4tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)4218
+;-----------------------------------------------------------------------------------------------------------------4219
+%macro CHROMA_VERT_S_16x4_AVX512 14220
+INIT_ZMM avx5124221
+cglobal interp_4tap_vert_%1_16x4, 5, 9, 124222
+ add r1d, r1d4223
+ add r3d, r3d4224
+ sub r0, r14225
+ shl r4d, 74226
+%ifdef PIC4227
+ lea r5, [tab_ChromaCoeffV_avx512]4228
+ mova m8, [r5 + r4]4229
+ mova m9, [r5 + r4 + mmsize]4230
+%else4231
+ lea r5, [tab_ChromaCoeffV_avx512 + r4]4232
+ mova m8, [r5]4233
+ mova m9, [r5 + mmsize]4234
+%endif4235
+4236
+%ifidn %1, sp4237
+ vbroadcasti32x4 m7, [INTERP_OFFSET_SP]4238
+ pxor m10, m104239
+ vbroadcasti32x8 m11, [pw_pixel_max]4240
+%endif4241
+ lea r7, [3 * r3]4242
+ lea r8, [3 * r1]4243
+ PROCESS_CHROMA_VERT_S_16x4_AVX512 %14244
+ RET4245
+%endmacro4246
+4247
+%if ARCH_X86_644248
+ CHROMA_VERT_S_16x4_AVX512 ss4249
+ CHROMA_VERT_S_16x4_AVX512 sp4250
+%endif4251
+%macro FILTER_VER_S_CHROMA_16xN_AVX512 24252
+INIT_ZMM avx5124253
+cglobal interp_4tap_vert_%1_16x%2, 5, 9, 124254
+ add r1d, r1d4255
+ add r3d, r3d4256
+ sub r0, r14257
+ shl r4d, 74258
+%ifdef PIC4259
+ lea r5, [tab_ChromaCoeffV_avx512]4260
+ mova m8, [r5 + r4]4261
+ mova m9, [r5 + r4 + mmsize]4262
+%else4263
+ lea r5, [tab_ChromaCoeffV_avx512 + r4]4264
+ mova m8, [r5]4265
+ mova m9, [r5 + mmsize]4266
+%endif4267
+4268
+%ifidn %1, sp4269
+ vbroadcasti32x4 m7, [INTERP_OFFSET_SP]4270
+ pxor m10, m104271
+ vbroadcasti32x8 m11, [pw_pixel_max]4272
+%endif4273
+ lea r7, [3 * r3]4274
+ lea r8, [3 * r1]4275
+%rep %2/4 - 14276
+ PROCESS_CHROMA_VERT_S_16x4_AVX512 %14277
+ lea r0, [r0 + 4 * r1]4278
+ lea r2, [r2 + 4 * r3]4279
+%endrep4280
+ PROCESS_CHROMA_VERT_S_16x4_AVX512 %14281
+ RET4282
+%endmacro4283
+4284
+%if ARCH_X86_644285
+ FILTER_VER_S_CHROMA_16xN_AVX512 ss, 84286
+ FILTER_VER_S_CHROMA_16xN_AVX512 ss, 124287
+ FILTER_VER_S_CHROMA_16xN_AVX512 ss, 164288
+ FILTER_VER_S_CHROMA_16xN_AVX512 ss, 244289
+ FILTER_VER_S_CHROMA_16xN_AVX512 ss, 324290
+ FILTER_VER_S_CHROMA_16xN_AVX512 ss, 644291
+ FILTER_VER_S_CHROMA_16xN_AVX512 sp, 84292
+ FILTER_VER_S_CHROMA_16xN_AVX512 sp, 124293
+ FILTER_VER_S_CHROMA_16xN_AVX512 sp, 164294
+ FILTER_VER_S_CHROMA_16xN_AVX512 sp, 244295
+ FILTER_VER_S_CHROMA_16xN_AVX512 sp, 324296
+ FILTER_VER_S_CHROMA_16xN_AVX512 sp, 644297
+%endif4298
+4299
+%macro PROCESS_CHROMA_VERT_S_24x8_AVX512 14300
+ movu ym1, [r0]4301
+ lea r6, [r0 + 2 * r1]4302
+ lea r8, [r0 + 4 * r1]4303
+ lea r9, [r8 + 2 * r1]4304
+4305
+ movu ym10, [r8]4306
+ movu ym3, [r0 + r1]4307
+ movu ym12, [r8 + r1]4308
+ vinserti32x8 m1, [r6], 14309
+ vinserti32x8 m10, [r9], 14310
+ vinserti32x8 m3, [r6 + r1], 14311
+ vinserti32x8 m12, [r9 + r1], 14312
+4313
+ punpcklwd m0, m1, m34314
+ punpcklwd m9, m10, m124315
+ pmaddwd m0, m164316
+ pmaddwd m9, m164317
+ punpckhwd m1, m34318
+ punpckhwd m10, m124319
+ pmaddwd m1, m164320
+ pmaddwd m10, m164321
+4322
+ movu ym4, [r0 + 2 * r1]4323
+ movu ym13, [r8 + 2 * r1]4324
+ vinserti32x8 m4, [r6 + 2 * r1], 14325
+ vinserti32x8 m13, [r9 + 2 * r1], 14326
+ punpcklwd m2, m3, m44327
+ punpcklwd m11, m12, m134328
+ pmaddwd m2, m164329
+ pmaddwd m11, m164330
+ punpckhwd m3, m44331
+ punpckhwd m12, m134332
+ pmaddwd m3, m164333
+ pmaddwd m12, m164334
+4335
+ movu ym5, [r0 + r10]4336
+ vinserti32x8 m5, [r6 + r10], 14337
+ movu ym14, [r8 + r10]4338
+ vinserti32x8 m14, [r9 + r10], 14339
+ punpcklwd m6, m4, m54340
+ punpcklwd m15, m13, m144341
+ pmaddwd m6, m174342
+ pmaddwd m15, m174343
+ paddd m0, m64344
+ paddd m9, m154345
+ punpckhwd m4, m54346
+ punpckhwd m13, m144347
+ pmaddwd m4, m174348
+ pmaddwd m13, m174349
+ paddd m1, m44350
+ paddd m10, m134351
+4352
+ movu ym4, [r0 + 4 * r1]4353
+ vinserti32x8 m4, [r6 + 4 * r1], 14354
+ movu ym13, [r8 + 4 * r1]4355
+ vinserti32x8 m13, [r9 + 4 * r1], 14356
+ punpcklwd m6, m5, m44357
+ punpcklwd m15, m14, m134358
+ pmaddwd m6, m174359
+ pmaddwd m15, m174360
+ paddd m2, m64361
+ paddd m11, m154362
+ punpckhwd m5, m44363
+ punpckhwd m14, m134364
+ pmaddwd m5, m174365
+ pmaddwd m14, m174366
+ paddd m3, m54367
+ paddd m12, m144368
+4369
+%ifidn %1,sp4370
+ paddd m0, m74371
+ paddd m1, m74372
+ paddd m2, m74373
+ paddd m3, m74374
+ paddd m9, m74375
+ paddd m10, m74376
+ paddd m11, m74377
+ paddd m12, m74378
+4379
+ psrad m0, INTERP_SHIFT_SP4380
+ psrad m1, INTERP_SHIFT_SP4381
+ psrad m2, INTERP_SHIFT_SP4382
+ psrad m3, INTERP_SHIFT_SP4383
+ psrad m9, INTERP_SHIFT_SP4384
+ psrad m10, INTERP_SHIFT_SP4385
+ psrad m11, INTERP_SHIFT_SP4386
+ psrad m12, INTERP_SHIFT_SP4387
+4388
+ packssdw m0, m14389
+ packssdw m2, m34390
+ packssdw m9, m104391
+ packssdw m11, m124392
+ CLIPW2 m0, m2, m18, m194393
+ CLIPW2 m9, m11, m18, m194394
+%else4395
+ psrad m0, 64396
+ psrad m1, 64397
+ psrad m2, 64398
+ psrad m3, 64399
+ psrad m9, 64400
+ psrad m10, 64401
+ psrad m11, 64402
+ psrad m12, 64403
+4404
+ packssdw m0, m14405
+ packssdw m2, m34406
+ packssdw m9, m104407
+ packssdw m11, m124408
+%endif4409
+4410
+ movu [r2], ym04411
+ movu [r2 + r3], ym24412
+ vextracti32x8 [r2 + 2 * r3], m0, 14413
+ vextracti32x8 [r2 + r7], m2, 14414
+ lea r11, [r2 + 4 * r3]4415
+ movu [r11], ym94416
+ movu [r11 + r3], ym114417
+ vextracti32x8 [r11 + 2 * r3], m9, 14418
+ vextracti32x8 [r11 + r7], m11, 14419
+4420
+ movu xm1, [r0 + mmsize/2]4421
+ vinserti32x4 m1, [r6 + mmsize/2], 14422
+ vinserti32x4 m1, [r8 + mmsize/2], 24423
+ vinserti32x4 m1, [r9 + mmsize/2], 34424
+ movu xm3, [r0 + r1 + mmsize/2]4425
+ vinserti32x4 m3, [r6 + r1 + mmsize/2], 14426
+ vinserti32x4 m3, [r8 + r1 + mmsize/2], 24427
+ vinserti32x4 m3, [r9 + r1 + mmsize/2], 34428
+ punpcklwd m0, m1, m34429
+ pmaddwd m0, m164430
+ punpckhwd m1, m34431
+ pmaddwd m1, m164432
+4433
+ movu xm4, [r0 + 2 * r1 + mmsize/2]4434
+ vinserti32x4 m4, [r6 + 2 * r1 + mmsize/2], 14435
+ vinserti32x4 m4, [r8 + 2 * r1 + mmsize/2], 24436
+ vinserti32x4 m4, [r9 + 2 * r1 + mmsize/2], 34437
+ punpcklwd m2, m3, m44438
+ pmaddwd m2, m164439
+ punpckhwd m3, m44440
+ pmaddwd m3, m164441
+4442
+ movu xm5, [r0 + r10 + mmsize/2]4443
+ vinserti32x4 m5, [r6 + r10 + mmsize/2], 14444
+ vinserti32x4 m5, [r8 + r10 + mmsize/2], 24445
+ vinserti32x4 m5, [r9 + r10 + mmsize/2], 34446
+ punpcklwd m6, m4, m54447
+ pmaddwd m6, m174448
+ paddd m0, m64449
+ punpckhwd m4, m54450
+ pmaddwd m4, m174451
+ paddd m1, m44452
+4453
+ movu xm4, [r0 + 4 * r1 + mmsize/2]4454
+ vinserti32x4 m4, [r6 + 4 * r1 + mmsize/2], 14455
+ vinserti32x4 m4, [r8 + 4 * r1 + mmsize/2], 24456
+ vinserti32x4 m4, [r9 + 4 * r1 + mmsize/2], 34457
+ punpcklwd m6, m5, m44458
+ pmaddwd m6, m174459
+ paddd m2, m64460
+ punpckhwd m5, m44461
+ pmaddwd m5, m174462
+ paddd m3, m54463
+4464
+%ifidn %1,sp4465
+ paddd m0, m74466
+ paddd m1, m74467
+ paddd m2, m74468
+ paddd m3, m74469
+4470
+ psrad m0, INTERP_SHIFT_SP4471
+ psrad m1, INTERP_SHIFT_SP4472
+ psrad m2, INTERP_SHIFT_SP4473
+ psrad m3, INTERP_SHIFT_SP4474
+4475
+ packssdw m0, m14476
+ packssdw m2, m34477
+ CLIPW2 m0, m2, m18, m194478
+%else4479
+ psrad m0, 64480
+ psrad m1, 64481
+ psrad m2, 64482
+ psrad m3, 64483
+4484
+ packssdw m0, m14485
+ packssdw m2, m34486
+%endif4487
+4488
+ movu [r2 + mmsize/2], xm04489
+ movu [r2 + r3 + mmsize/2], xm24490
+ vextracti32x4 [r2 + 2 * r3 + mmsize/2], m0, 14491
+ vextracti32x4 [r2 + r7 + mmsize/2], m2, 14492
+ lea r2, [r2 + 4 * r3]4493
+ vextracti32x4 [r2 + mmsize/2], m0, 24494
+ vextracti32x4 [r2 + r3 + mmsize/2], m2, 24495
+ vextracti32x4 [r2 + 2 * r3 + mmsize/2], m0, 34496
+ vextracti32x4 [r2 + r7 + mmsize/2], m2, 34497
+%endmacro4498
+%macro FILTER_VER_S_CHROMA_24xN_AVX512 24499
+INIT_ZMM avx5124500
+cglobal interp_4tap_vert_%1_24x%2, 5, 12, 204501
+ add r1d, r1d4502
+ add r3d, r3d4503
+ sub r0, r14504
+ shl r4d, 74505
+%ifdef PIC4506
+ lea r5, [tab_ChromaCoeffV_avx512]4507
+ mova m16, [r5 + r4]4508
+ mova m17, [r5 + r4 + mmsize]4509
+%else4510
+ lea r5, [tab_ChromaCoeffV_avx512 + r4]4511
+ mova m16, [r5]4512
+ mova m17, [r5 + mmsize]4513
+%endif4514
+%ifidn %1, sp4515
+ vbroadcasti32x4 m7, [INTERP_OFFSET_SP]4516
+ pxor m18, m184517
+ vbroadcasti32x8 m19, [pw_pixel_max]4518
+%endif4519
+ lea r10, [3 * r1]4520
+ lea r7, [3 * r3]4521
+%rep %2/8 - 14522
+ PROCESS_CHROMA_VERT_S_24x8_AVX512 %14523
+ lea r0, [r8 + 4 * r1]4524
+ lea r2, [r2 + 4 * r3]4525
+%endrep4526
+ PROCESS_CHROMA_VERT_S_24x8_AVX512 %14527
+ RET4528
+%endmacro4529
+%if ARCH_X86_644530
+ FILTER_VER_S_CHROMA_24xN_AVX512 ss, 324531
+ FILTER_VER_S_CHROMA_24xN_AVX512 ss, 644532
+ FILTER_VER_S_CHROMA_24xN_AVX512 sp, 324533
+ FILTER_VER_S_CHROMA_24xN_AVX512 sp, 644534
+%endif4535
+4536
+%macro PROCESS_CHROMA_VERT_S_32x4_AVX512 14537
+ movu m1, [r0]4538
+ lea r6, [r0 + 2 * r1]4539
+ movu m10, [r6]4540
+ movu m3, [r0 + r1]4541
+ movu m12, [r6 + r1]4542
+ punpcklwd m0, m1, m34543
+ punpcklwd m9, m10, m124544
+ pmaddwd m0, m164545
+ pmaddwd m9, m164546
+ punpckhwd m1, m34547
+ punpckhwd m10, m124548
+ pmaddwd m1, m164549
+ pmaddwd m10, m164550
+ movu m4, [r0 + 2 * r1]4551
+ movu m13, [r6 + 2 * r1]4552
+ punpcklwd m2, m3, m44553
+ punpcklwd m11, m12, m134554
+ pmaddwd m2, m164555
+ pmaddwd m11, m164556
+ punpckhwd m3, m44557
+ punpckhwd m12, m134558
+ pmaddwd m3, m164559
+ pmaddwd m12, m164560
+4561
+ movu m5, [r0 + r7]4562
+ movu m14, [r6 + r7]4563
+ punpcklwd m6, m4, m54564
+ punpcklwd m15, m13, m144565
+ pmaddwd m6, m174566
+ pmaddwd m15, m174567
+ paddd m0, m64568
+ paddd m9, m154569
+ punpckhwd m4, m54570
+ punpckhwd m13, m144571
+ pmaddwd m4, m174572
+ pmaddwd m13, m174573
+ paddd m1, m44574
+ paddd m10, m134575
+4576
+ movu m4, [r0 + 4 * r1]4577
+ movu m13, [r6 + 4 * r1]4578
+ punpcklwd m6, m5, m44579
+ punpcklwd m15, m14, m134580
+ pmaddwd m6, m174581
+ pmaddwd m15, m174582
+ paddd m2, m64583
+ paddd m11, m154584
+ punpckhwd m5, m44585
+ punpckhwd m14, m134586
+ pmaddwd m5, m174587
+ pmaddwd m14, m174588
+ paddd m3, m54589
+ paddd m12, m144590
+%ifidn %1,sp4591
+ paddd m0, m74592
+ paddd m1, m74593
+ paddd m2, m74594
+ paddd m3, m74595
+ paddd m9, m74596
+ paddd m10, m74597
+ paddd m11, m74598
+ paddd m12, m74599
+4600
+ psrad m0, INTERP_SHIFT_SP4601
+ psrad m1, INTERP_SHIFT_SP4602
+ psrad m2, INTERP_SHIFT_SP4603
+ psrad m3, INTERP_SHIFT_SP4604
+ psrad m9, INTERP_SHIFT_SP4605
+ psrad m10, INTERP_SHIFT_SP4606
+ psrad m11, INTERP_SHIFT_SP4607
+ psrad m12, INTERP_SHIFT_SP4608
+4609
+ packssdw m0, m14610
+ packssdw m2, m34611
+ packssdw m9, m104612
+ packssdw m11, m124613
+ CLIPW2 m0, m2, m18, m194614
+ CLIPW2 m9, m11, m18, m194615
+%else4616
+ psrad m0, 64617
+ psrad m1, 64618
+ psrad m2, 64619
+ psrad m3, 64620
+ psrad m9, 64621
+ psrad m10, 64622
+ psrad m11, 64623
+ psrad m12, 64624
+4625
+ packssdw m0, m14626
+ packssdw m2, m34627
+ packssdw m9, m104628
+ packssdw m11, m124629
+%endif4630
+4631
+ movu [r2], m04632
+ movu [r2 + r3], m24633
+ movu [r2 + 2 * r3], m94634
+ movu [r2 + r8], m114635
+%endmacro4636
+;-----------------------------------------------------------------------------------------------------------------4637
+; void interp_4tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)4638
+;-----------------------------------------------------------------------------------------------------------------4639
+%macro FILTER_VER_S_CHROMA_32xN_AVX512 24640
+INIT_ZMM avx5124641
+cglobal interp_4tap_vert_%1_32x%2, 5, 9, 204642
+ add r1d, r1d4643
+ add r3d, r3d4644
+ sub r0, r14645
+ shl r4d, 74646
+%ifdef PIC4647
+ lea r5, [tab_ChromaCoeffV_avx512]4648
+ mova m16, [r5 + r4]4649
+ mova m17, [r5 + r4 + mmsize]4650
+%else4651
+ lea r5, [tab_ChromaCoeffV_avx512 + r4]4652
+ mova m16, [r5]4653
+ mova m17, [r5 + mmsize]4654
+%endif4655
+ lea r7, [3 * r1]4656
+ lea r8, [3 * r3]4657
+%ifidn %1, sp4658
+ vbroadcasti32x4 m7, [INTERP_OFFSET_SP]4659
+ pxor m18, m184660
+ vbroadcasti32x8 m19, [pw_pixel_max]4661
+%endif4662
+4663
+%rep %2/4 - 14664
+ PROCESS_CHROMA_VERT_S_32x4_AVX512 %14665
+ lea r0, [r0 + 4 * r1]4666
+ lea r2, [r2 + 4 * r3]4667
+%endrep4668
+ PROCESS_CHROMA_VERT_S_32x4_AVX512 %14669
+ RET4670
+%endmacro4671
+%if ARCH_X86_644672
+ FILTER_VER_S_CHROMA_32xN_AVX512 ss, 84673
+ FILTER_VER_S_CHROMA_32xN_AVX512 ss, 164674
+ FILTER_VER_S_CHROMA_32xN_AVX512 ss, 244675
+ FILTER_VER_S_CHROMA_32xN_AVX512 ss, 324676
+ FILTER_VER_S_CHROMA_32xN_AVX512 ss, 484677
+ FILTER_VER_S_CHROMA_32xN_AVX512 ss, 644678
+ FILTER_VER_S_CHROMA_32xN_AVX512 sp, 84679
+ FILTER_VER_S_CHROMA_32xN_AVX512 sp, 164680
+ FILTER_VER_S_CHROMA_32xN_AVX512 sp, 244681
+ FILTER_VER_S_CHROMA_32xN_AVX512 sp, 324682
+ FILTER_VER_S_CHROMA_32xN_AVX512 sp, 484683
+ FILTER_VER_S_CHROMA_32xN_AVX512 sp, 644684
+%endif4685
+%macro PROCESS_CHROMA_VERT_S_48x4_AVX512 14686
+ movu m1, [r0]4687
+ lea r6, [r0 + 2 * r1]4688
+ movu m10, [r6]4689
+ movu m3, [r0 + r1]4690
+ movu m12, [r6 + r1]4691
+ punpcklwd m0, m1, m34692
+ punpcklwd m9, m10, m124693
+ pmaddwd m0, m164694
+ pmaddwd m9, m164695
+ punpckhwd m1, m34696
+ punpckhwd m10, m124697
+ pmaddwd m1, m164698
+ pmaddwd m10, m164699
+4700
+ movu m4, [r0 + 2 * r1]4701
+ movu m13, [r6 + 2 * r1]4702
+ punpcklwd m2, m3, m44703
+ punpcklwd m11, m12, m134704
+ pmaddwd m2, m164705
+ pmaddwd m11, m164706
+ punpckhwd m3, m44707
+ punpckhwd m12, m134708
+ pmaddwd m3, m164709
+ pmaddwd m12, m164710
+4711
+ movu m5, [r0 + r7]4712
+ movu m14, [r6 + r7]4713
+ punpcklwd m6, m4, m54714
+ punpcklwd m15, m13, m144715
+ pmaddwd m6, m174716
+ pmaddwd m15, m174717
+ paddd m0, m64718
+ paddd m9, m154719
+ punpckhwd m4, m54720
+ punpckhwd m13, m144721
+ pmaddwd m4, m174722
+ pmaddwd m13, m174723
+ paddd m1, m44724
+ paddd m10, m134725
+4726
+ movu m4, [r0 + 4 * r1]4727
+ movu m13, [r6 + 4 * r1]4728
+ punpcklwd m6, m5, m44729
+ punpcklwd m15, m14, m134730
+ pmaddwd m6, m174731
+ pmaddwd m15, m174732
+ paddd m2, m64733
+ paddd m11, m154734
+ punpckhwd m5, m44735
+ punpckhwd m14, m134736
+ pmaddwd m5, m174737
+ pmaddwd m14, m174738
+ paddd m3, m54739
+ paddd m12, m144740
+4741
+%ifidn %1,sp4742
+ paddd m0, m74743
+ paddd m1, m74744
+ paddd m2, m74745
+ paddd m3, m74746
+ paddd m9, m74747
+ paddd m10, m74748
+ paddd m11, m74749
+ paddd m12, m74750
+4751
+ psrad m0, INTERP_SHIFT_SP4752
+ psrad m1, INTERP_SHIFT_SP4753
+ psrad m2, INTERP_SHIFT_SP4754
+ psrad m3, INTERP_SHIFT_SP4755
+ psrad m9, INTERP_SHIFT_SP4756
+ psrad m10, INTERP_SHIFT_SP4757
+ psrad m11, INTERP_SHIFT_SP4758
+ psrad m12, INTERP_SHIFT_SP4759
+4760
+ packssdw m0, m14761
+ packssdw m2, m34762
+ packssdw m9, m104763
+ packssdw m11, m124764
+ CLIPW2 m0, m2, m18, m194765
+ CLIPW2 m9, m11, m18, m194766
+%else4767
+ psrad m0, 64768
+ psrad m1, 64769
+ psrad m2, 64770
+ psrad m3, 64771
+ psrad m9, 64772
+ psrad m10, 64773
+ psrad m11, 64774
+ psrad m12, 64775
+ packssdw m0, m14776
+ packssdw m2, m34777
+ packssdw m9, m104778
+ packssdw m11, m124779
+%endif4780
+4781
+ movu [r2], m04782
+ movu [r2 + r3], m24783
+ movu [r2 + 2 * r3], m94784
+ movu [r2 + r8], m114785
+4786
+ movu ym1, [r0 + mmsize]4787
+ vinserti32x8 m1, [r6 + mmsize], 14788
+ movu ym3, [r0 + r1 + mmsize]4789
+ vinserti32x8 m3, [r6 + r1 + mmsize], 14790
+ punpcklwd m0, m1, m34791
+ pmaddwd m0, m164792
+ punpckhwd m1, m34793
+ pmaddwd m1, m164794
+4795
+ movu ym4, [r0 + 2 * r1 + mmsize]4796
+ vinserti32x8 m4, [r6 + 2 * r1 + mmsize], 14797
+ punpcklwd m2, m3, m44798
+ pmaddwd m2, m164799
+ punpckhwd m3, m44800
+ pmaddwd m3, m164801
+4802
+ movu ym5, [r0 + r7 + mmsize]4803
+ vinserti32x8 m5, [r6 + r7 + mmsize], 14804
+ punpcklwd m6, m4, m54805
+ pmaddwd m6, m174806
+ paddd m0, m64807
+ punpckhwd m4, m54808
+ pmaddwd m4, m174809
+ paddd m1, m44810
+4811
+ movu ym4, [r0 + 4 * r1 + mmsize]4812
+ vinserti32x8 m4, [r6 + 4 * r1 + mmsize], 14813
+ punpcklwd m6, m5, m44814
+ pmaddwd m6, m174815
+ paddd m2, m64816
+ punpckhwd m5, m44817
+ pmaddwd m5, m174818
+ paddd m3, m54819
+4820
+%ifidn %1,sp4821
+ paddd m0, m74822
+ paddd m1, m74823
+ paddd m2, m74824
+ paddd m3, m74825
+4826
+ psrad m0, INTERP_SHIFT_SP4827
+ psrad m1, INTERP_SHIFT_SP4828
+ psrad m2, INTERP_SHIFT_SP4829
+ psrad m3, INTERP_SHIFT_SP4830
+ packssdw m0, m14831
+ packssdw m2, m34832
+ CLIPW2 m0, m2, m18, m194833
+%else4834
+ psrad m0, 64835
+ psrad m1, 64836
+ psrad m2, 64837
+ psrad m3, 64838
+ packssdw m0, m14839
+ packssdw m2, m34840
+%endif4841
+4842
+ movu [r2 + mmsize], ym04843
+ movu [r2 + r3 + mmsize], ym24844
+ vextracti32x8 [r2 + 2 * r3 + mmsize], m0, 14845
+ vextracti32x8 [r2 + r8 + mmsize], m2, 14846
+%endmacro4847
+%macro CHROMA_VERT_S_48x4_AVX512 14848
+INIT_ZMM avx5124849
+cglobal interp_4tap_vert_%1_48x64, 5, 9, 204850
+ add r1d, r1d4851
+ add r3d, r3d4852
+ sub r0, r14853
+ shl r4d, 74854
+%ifdef PIC4855
+ lea r5, [tab_ChromaCoeffV_avx512]4856
+ mova m16, [r5 + r4]4857
+ mova m17, [r5 + r4 + mmsize]4858
+%else4859
+ lea r5, [tab_ChromaCoeffV_avx512 + r4]4860
+ mova m16, [r5]4861
+ mova m17, [r5 + mmsize]4862
+%endif4863
+ lea r7, [3 * r1]4864
+ lea r8, [3 * r3]4865
+%ifidn %1, sp4866
+ vbroadcasti32x4 m7, [INTERP_OFFSET_SP]4867
+ pxor m18, m184868
+ vbroadcasti32x8 m19, [pw_pixel_max]4869
+%endif4870
+%rep 154871
+ PROCESS_CHROMA_VERT_S_48x4_AVX512 %14872
+ lea r0, [r0 + 4 * r1]4873
+ lea r2, [r2 + 4 * r3]4874
+%endrep4875
+ PROCESS_CHROMA_VERT_S_48x4_AVX512 %14876
+ RET4877
+%endmacro4878
+4879
+%if ARCH_X86_644880
+ CHROMA_VERT_S_48x4_AVX512 sp4881
+ CHROMA_VERT_S_48x4_AVX512 ss4882
+%endif4883
+%macro PROCESS_CHROMA_VERT_S_64x2_AVX512 14884
+ movu m1, [r0]4885
+ movu m3, [r0 + r1]4886
+ punpcklwd m0, m1, m34887
+ pmaddwd m0, m154888
+ punpckhwd m1, m34889
+ pmaddwd m1, m154890
+4891
+ movu m9, [r0 + mmsize]4892
+ movu m11, [r0 + r1 + mmsize]4893
+ punpcklwd m8, m9, m114894
+ pmaddwd m8, m154895
+ punpckhwd m9, m114896
+ pmaddwd m9, m154897
+ movu m4, [r0 + 2 * r1]4898
+ punpcklwd m2, m3, m44899
+ pmaddwd m2, m154900
+ punpckhwd m3, m44901
+ pmaddwd m3, m154902
+ movu m12, [r0 + 2 * r1 + mmsize]4903
+ punpcklwd m10, m11, m124904
+ pmaddwd m10, m154905
+ punpckhwd m11, m124906
+ pmaddwd m11, m154907
+4908
+ lea r0, [r0 + 2 * r1]4909
+ movu m5, [r0 + r1]4910
+ punpcklwd m6, m4, m54911
+ pmaddwd m6, m164912
+ paddd m0, m64913
+ punpckhwd m4, m54914
+ pmaddwd m4, m164915
+ paddd m1, m44916
+4917
+ movu m13, [r0 + r1 + mmsize]4918
+ punpcklwd m14, m12, m134919
+ pmaddwd m14, m164920
+ paddd m8, m144921
+ punpckhwd m12, m134922
+ pmaddwd m12, m164923
+ paddd m9, m124924
+4925
+ movu m4, [r0 + 2 * r1]4926
+ punpcklwd m6, m5, m44927
+ pmaddwd m6, m164928
+ paddd m2, m64929
+ punpckhwd m5, m44930
+ pmaddwd m5, m164931
+ paddd m3, m54932
+4933
+ movu m12, [r0 + 2 * r1 + mmsize]4934
+ punpcklwd m14, m13, m124935
+ pmaddwd m14, m164936
+ paddd m10, m144937
+ punpckhwd m13, m124938
+ pmaddwd m13, m164939
+ paddd m11, m134940
+4941
+%ifidn %1,sp4942
+ paddd m0, m74943
+ paddd m1, m74944
+ paddd m2, m74945
+ paddd m3, m74946
+ paddd m8, m74947
+ paddd m9, m74948
+ paddd m10, m74949
+ paddd m11, m74950
+4951
+ psrad m0, INTERP_SHIFT_SP4952
+ psrad m1, INTERP_SHIFT_SP4953
+ psrad m2, INTERP_SHIFT_SP4954
+ psrad m3, INTERP_SHIFT_SP4955
+ psrad m8, INTERP_SHIFT_SP4956
+ psrad m9, INTERP_SHIFT_SP4957
+ psrad m10, INTERP_SHIFT_SP4958
+ psrad m11, INTERP_SHIFT_SP4959
+4960
+ packssdw m0, m14961
+ packssdw m2, m34962
+ packssdw m8, m94963
+ packssdw m10, m114964
+ CLIPW2 m0, m2, m17, m184965
+ CLIPW2 m8, m10, m17, m184966
+%else4967
+ psrad m0, 64968
+ psrad m1, 64969
+ psrad m2, 64970
+ psrad m3, 64971
+ psrad m8, 64972
+ psrad m9, 64973
+ psrad m10, 64974
+ psrad m11, 64975
+4976
+ packssdw m0, m14977
+ packssdw m2, m34978
+ packssdw m8, m94979
+ packssdw m10, m114980
+%endif4981
+4982
+ movu [r2], m04983
+ movu [r2 + r3], m24984
+ movu [r2 + mmsize], m84985
+ movu [r2 + r3 + mmsize], m104986
+%endmacro4987
+;-----------------------------------------------------------------------------------------------------------------4988
+; void interp_4tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)4989
+;-----------------------------------------------------------------------------------------------------------------4990
+%macro FILTER_VER_S_CHROMA_64xN_AVX512 24991
+INIT_ZMM avx5124992
+cglobal interp_4tap_vert_%1_64x%2, 5, 7, 194993
+ add r1d, r1d4994
+ add r3d, r3d4995
+ sub r0, r14996
+ shl r4d, 74997
+%ifdef PIC4998
+ lea r5, [tab_ChromaCoeffV_avx512]4999
+ mova m15, [r5 + r4]5000
+ mova m16, [r5 + r4 + mmsize]5001
+%else5002
+ lea r5, [tab_ChromaCoeffV_avx512 + r4]5003
+ mova m15, [r5]5004
+ mova m16, [r5 + mmsize]5005
+%endif5006
+%ifidn %1, sp5007
+ vbroadcasti32x4 m7, [INTERP_OFFSET_SP]5008
+ pxor m17, m175009
+ vbroadcasti32x8 m18, [pw_pixel_max]5010
+%endif5011
+%rep %2/2 - 15012
+ PROCESS_CHROMA_VERT_S_64x2_AVX512 %15013
+ lea r2, [r2 + 2 * r3]5014
+%endrep5015
+ PROCESS_CHROMA_VERT_S_64x2_AVX512 %15016
+ RET5017
+%endmacro5018
+5019
+%if ARCH_X86_645020
+ FILTER_VER_S_CHROMA_64xN_AVX512 ss, 165021
+ FILTER_VER_S_CHROMA_64xN_AVX512 ss, 325022
+ FILTER_VER_S_CHROMA_64xN_AVX512 ss, 485023
+ FILTER_VER_S_CHROMA_64xN_AVX512 ss, 645024
+ FILTER_VER_S_CHROMA_64xN_AVX512 sp, 165025
+ FILTER_VER_S_CHROMA_64xN_AVX512 sp, 325026
+ FILTER_VER_S_CHROMA_64xN_AVX512 sp, 485027
+ FILTER_VER_S_CHROMA_64xN_AVX512 sp, 645028
+%endif5029
+;-------------------------------------------------------------------------------------------------------------5030
+; avx512 chroma_vsp and chroma_vss code end5031
+;-------------------------------------------------------------------------------------------------------------5032
+;-------------------------------------------------------------------------------------------------------------5033
+;ipfilter_chroma_avx512 code end5034
+;-------------------------------------------------------------------------------------------------------------5035
+;-------------------------------------------------------------------------------------------------------------5036
+;ipfilter_luma_avx512 code start5037
+;-------------------------------------------------------------------------------------------------------------5038
+%macro PROCESS_IPFILTER_LUMA_PP_8x4_AVX512 05039
+ ; register map5040
+ ; m0 , m1, m2, m3 - interpolate coeff5041
+ ; m4 , m5 load shuffle order table5042
+ ; m6 - pd_325043
+ ; m7 - zero5044
+ ; m8 - pw_pixel_max5045
+ ; m9 - store shuffle order table5046
+5047
+ movu xm10, [r0]5048
+ movu xm11, [r0 + 8]5049
+ movu xm12, [r0 + 16]5050
+5051
+ vinserti32x4 m10, [r0 + r1], 15052
+ vinserti32x4 m11, [r0 + r1 + 8], 15053
+ vinserti32x4 m12, [r0 + r1 + 16], 15054
+5055
+ vinserti32x4 m10, [r0 + 2 * r1], 25056
+ vinserti32x4 m11, [r0 + 2 * r1 + 8], 25057
+ vinserti32x4 m12, [r0 + 2 * r1 + 16], 25058
+5059
+ vinserti32x4 m10, [r0 + r6], 35060
+ vinserti32x4 m11, [r0 + r6 + 8], 35061
+ vinserti32x4 m12, [r0 + r6 + 16], 35062
+5063
+ pshufb m13, m10, m55064
+ pshufb m10, m45065
+ pshufb m14, m11, m55066
+ pshufb m11, m45067
+ pshufb m15, m12, m55068
+ pshufb m12, m45069
+5070
+ pmaddwd m10, m05071
+ pmaddwd m13, m15072
+ paddd m10, m135073
+ pmaddwd m13, m14, m35074
+ pmaddwd m16, m11, m25075
+ paddd m13, m165076
+ paddd m10, m135077
+ paddd m10, m65078
+ psrad m10, INTERP_SHIFT_PP5079
+5080
+ pmaddwd m11, m05081
+ pmaddwd m14, m15082
+ paddd m11, m145083
+ pmaddwd m15, m35084
+ pmaddwd m12, m25085
+ paddd m12, m155086
+ paddd m11, m125087
+ paddd m11, m65088
+ psrad m11, INTERP_SHIFT_PP5089
+5090
+ packusdw m10, m115091
+ CLIPW m10, m7, m85092
+ pshufb m10, m95093
+ movu [r2], xm105094
+ vextracti32x4 [r2 + r3], m10, 15095
+ vextracti32x4 [r2 + 2 * r3], m10, 25096
+ vextracti32x4 [r2 + r7], m10, 35097
+%endmacro5098
+5099
+%macro PROCESS_IPFILTER_LUMA_PP_16x4_AVX512 05100
+ ; register map5101
+ ; m0 , m1, m2, m3 - interpolate coeff5102
+ ; m4 , m5 load shuffle order table5103
+ ; m6 - pd_325104
+ ; m7 - zero5105
+ ; m8 - pw_pixel_max5106
+ ; m9 - store shuffle order table5107
+5108
+ movu ym10, [r0]5109
+ vinserti32x8 m10, [r0 + r1], 15110
+ movu ym11, [r0 + 8]5111
+ vinserti32x8 m11, [r0 + r1 + 8], 15112
+ movu ym12, [r0 + 16]5113
+ vinserti32x8 m12, [r0 + r1 + 16], 15114
+5115
+ pshufb m13, m10, m55116
+ pshufb m10, m45117
+ pshufb m14, m11, m55118
+ pshufb m11, m45119
+ pshufb m15, m12, m55120
+ pshufb m12, m45121
+5122
+ pmaddwd m10, m05123
+ pmaddwd m13, m15124
+ paddd m10, m135125
+ pmaddwd m13, m14, m35126
+ pmaddwd m16, m11, m25127
+ paddd m13, m165128
+ paddd m10, m135129
+ paddd m10, m65130
+ psrad m10, INTERP_SHIFT_PP5131
+5132
+ pmaddwd m11, m05133
+ pmaddwd m14, m15134
+ paddd m11, m145135
+ pmaddwd m15, m35136
+ pmaddwd m12, m25137
+ paddd m12, m155138
+ paddd m11, m125139
+ paddd m11, m65140
+ psrad m11, INTERP_SHIFT_PP5141
+5142
+ packusdw m10, m115143
+ CLIPW m10, m7, m85144
+ pshufb m10, m95145
+ movu [r2], ym105146
+ vextracti32x8 [r2 + r3], m10, 15147
+5148
+ movu ym10, [r0 + 2 * r1]5149
+ vinserti32x8 m10, [r0 + r6], 15150
+ movu ym11, [r0 + 2 * r1 + 8]5151
+ vinserti32x8 m11, [r0 + r6 + 8], 15152
+ movu ym12, [r0 + 2 * r1 + 16]5153
+ vinserti32x8 m12, [r0 + r6 + 16], 15154
+5155
+ pshufb m13, m10, m55156
+ pshufb m10, m45157
+ pshufb m14, m11, m55158
+ pshufb m11, m45159
+ pshufb m15, m12, m55160
+ pshufb m12, m45161
+5162
+ pmaddwd m10, m05163
+ pmaddwd m13, m15164
+ paddd m10, m135165
+ pmaddwd m13, m14, m35166
+ pmaddwd m16, m11, m25167
+ paddd m13, m165168
+ paddd m10, m135169
+ paddd m10, m65170
+ psrad m10, INTERP_SHIFT_PP5171
+5172
+ pmaddwd m11, m05173
+ pmaddwd m14, m15174
+ paddd m11, m145175
+ pmaddwd m14, m15, m35176
+ pmaddwd m16, m12, m25177
+ paddd m14, m165178
+ paddd m11, m145179
+ paddd m11, m65180
+ psrad m11, INTERP_SHIFT_PP5181
+5182
+ packusdw m10, m115183
+ CLIPW m10, m7, m85184
+ pshufb m10, m95185
+ movu [r2 + 2 * r3], ym105186
+ vextracti32x8 [r2 + r7], m10, 15187
+%endmacro5188
+5189
+%macro PROCESS_IPFILTER_LUMA_PP_24x4_AVX512 05190
+ ; register map5191
+ ; m0 , m1, m2, m3 - interpolate coeff5192
+ ; m4 , m5 load shuffle order table5193
+ ; m6 - pd_325194
+ ; m7 - zero5195
+ ; m8 - pw_pixel_max5196
+ ; m9 - store shuffle order table5197
+5198
+ PROCESS_IPFILTER_LUMA_PP_16x4_AVX5125199
+5200
+ movu xm10, [r0 + mmsize/2]5201
+ movu xm11, [r0 + mmsize/2 + 8]5202
+ movu xm12, [r0 + mmsize/2 + 16]5203
+5204
+ vinserti32x4 m10, [r0 + r1 + mmsize/2], 15205
+ vinserti32x4 m11, [r0 + r1 + mmsize/2 + 8], 15206
+ vinserti32x4 m12, [r0 + r1 + mmsize/2 + 16], 15207
+5208
+ vinserti32x4 m10, [r0 + 2 * r1 + mmsize/2], 25209
+ vinserti32x4 m11, [r0 + 2 * r1 + mmsize/2 + 8], 25210
+ vinserti32x4 m12, [r0 + 2 * r1 + mmsize/2 + 16], 25211
+5212
+ vinserti32x4 m10, [r0 + r6 + mmsize/2], 35213
+ vinserti32x4 m11, [r0 + r6 + mmsize/2 + 8], 35214
+ vinserti32x4 m12, [r0 + r6 + mmsize/2 + 16], 35215
+5216
+ pshufb m13, m10, m55217
+ pshufb m10, m45218
+ pshufb m14, m11, m55219
+ pshufb m11, m45220
+ pshufb m15, m12, m55221
+ pshufb m12, m45222
+5223
+ pmaddwd m10, m05224
+ pmaddwd m13, m15225
+ paddd m10, m135226
+ pmaddwd m13, m14, m35227
+ pmaddwd m16, m11, m25228
+ paddd m13, m165229
+ paddd m10, m135230
+ paddd m10, m65231
+ psrad m10, INTERP_SHIFT_PP5232
+5233
+ pmaddwd m11, m05234
+ pmaddwd m14, m15235
+ paddd m11, m145236
+ pmaddwd m15, m35237
+ pmaddwd m12, m25238
+ paddd m12, m155239
+ paddd m11, m125240
+ paddd m11, m65241
+ psrad m11, INTERP_SHIFT_PP5242
+5243
+ packusdw m10, m115244
+ CLIPW m10, m7, m85245
+ pshufb m10, m95246
+ movu [r2 + mmsize/2], xm105247
+ vextracti32x4 [r2 + r3 + mmsize/2], m10, 15248
+ vextracti32x4 [r2 + 2 * r3 + mmsize/2], m10, 25249
+ vextracti32x4 [r2 + r7 + mmsize/2], m10, 35250
+%endmacro5251
+5252
+%macro PROCESS_IPFILTER_LUMA_PP_32x2_AVX512 05253
+ ; register map5254
+ ; m0 , m1, m2, m3 - interpolate coeff5255
+ ; m4 , m5 load shuffle order table5256
+ ; m6 - pd_325257
+ ; m7 - zero5258
+ ; m8 - pw_pixel_max5259
+ ; m9 - store shuffle order table5260
+5261
+ movu m10, [r0]5262
+ movu m11, [r0 + 8]5263
+ movu m12, [r0 + 16]5264
+5265
+ pshufb m13, m10, m55266
+ pshufb m10, m45267
+ pshufb m14, m11, m55268
+ pshufb m11, m45269
+ pshufb m15, m12, m55270
+ pshufb m12, m45271
+5272
+ pmaddwd m10, m05273
+ pmaddwd m13, m15274
+ paddd m10, m135275
+ pmaddwd m13, m14, m35276
+ pmaddwd m16, m11, m25277
+ paddd m13, m165278
+ paddd m10, m135279
+ paddd m10, m65280
+ psrad m10, INTERP_SHIFT_PP5281
+5282
+ pmaddwd m11, m05283
+ pmaddwd m14, m15284
+ paddd m11, m145285
+ pmaddwd m15, m35286
+ pmaddwd m12, m25287
+ paddd m12, m155288
+ paddd m11, m125289
+ paddd m11, m65290
+ psrad m11, INTERP_SHIFT_PP5291
+5292
+ packusdw m10, m115293
+ CLIPW m10, m7, m85294
+ pshufb m10, m95295
+ movu [r2], m105296
+5297
+ movu m10, [r0 + r1]5298
+ movu m11, [r0 + r1 + 8]5299
+ movu m12, [r0 + r1 + 16]5300
+5301
+ pshufb m13, m10, m55302
+ pshufb m10, m45303
+ pshufb m14, m11, m55304
+ pshufb m11, m45305
+ pshufb m15, m12, m55306
+ pshufb m12, m45307
+5308
+ pmaddwd m10, m05309
+ pmaddwd m13, m15310
+ paddd m10, m135311
+ pmaddwd m13, m14, m35312
+ pmaddwd m16, m11, m25313
+ paddd m13, m165314
+ paddd m10, m135315
+ paddd m10, m65316
+ psrad m10, INTERP_SHIFT_PP5317
+5318
+ pmaddwd m11, m05319
+ pmaddwd m14, m15320
+ paddd m11, m145321
+ pmaddwd m14, m15, m35322
+ pmaddwd m16, m12, m25323
+ paddd m14, m165324
+ paddd m11, m145325
+ paddd m11, m65326
+ psrad m11, INTERP_SHIFT_PP5327
+5328
+ packusdw m10, m115329
+ CLIPW m10, m7, m85330
+ pshufb m10, m95331
+ movu [r2 + r3], m105332
+%endmacro5333
+5334
+%macro PROCESS_IPFILTER_LUMA_PP_48x4_AVX512 05335
+ ; register map5336
+ ; m0 , m1, m2, m3 - interpolate coeff5337
+ ; m4 , m5 load shuffle order table5338
+ ; m6 - pd_325339
+ ; m7 - zero5340
+ ; m8 - pw_pixel_max5341
+ ; m9 - store shuffle order table5342
+5343
+ movu m10, [r0]5344
+ movu m11, [r0 + 8]5345
+ movu m12, [r0 + 16]5346
+5347
+ pshufb m13, m10, m55348
+ pshufb m10, m45349
+ pshufb m14, m11, m55350
+ pshufb m11, m45351
+ pshufb m15, m12, m55352
+ pshufb m12, m45353
+5354
+ pmaddwd m10, m05355
+ pmaddwd m13, m15356
+ paddd m10, m135357
+ pmaddwd m13, m14, m35358
+ pmaddwd m16, m11, m25359
+ paddd m13, m165360
+ paddd m10, m135361
+ paddd m10, m65362
+ psrad m10, INTERP_SHIFT_PP5363
+5364
+ pmaddwd m11, m05365
+ pmaddwd m14, m15366
+ paddd m11, m145367
+ pmaddwd m15, m35368
+ pmaddwd m12, m25369
+ paddd m12, m155370
+ paddd m11, m125371
+ paddd m11, m65372
+ psrad m11, INTERP_SHIFT_PP5373
+5374
+ packusdw m10, m115375
+ CLIPW m10, m7, m85376
+ pshufb m10, m95377
+ movu [r2], m105378
+5379
+ movu m10, [r0 + r1]5380
+ movu m11, [r0 + r1 + 8]5381
+ movu m12, [r0 + r1 + 16]5382
+5383
+ pshufb m13, m10, m55384
+ pshufb m10, m45385
+ pshufb m14, m11, m55386
+ pshufb m11, m45387
+ pshufb m15, m12, m55388
+ pshufb m12, m45389
+5390
+ pmaddwd m10, m05391
+ pmaddwd m13, m15392
+ paddd m10, m135393
+ pmaddwd m13, m14, m35394
+ pmaddwd m16, m11, m25395
+ paddd m13, m165396
+ paddd m10, m135397
+ paddd m10, m65398
+ psrad m10, INTERP_SHIFT_PP5399
+5400
+ pmaddwd m11, m05401
+ pmaddwd m14, m15402
+ paddd m11, m145403
+ pmaddwd m14, m15, m35404
+ pmaddwd m16, m12, m25405
+ paddd m14, m165406
+ paddd m11, m145407
+ paddd m11, m65408
+ psrad m11, INTERP_SHIFT_PP5409
+5410
+ packusdw m10, m115411
+ CLIPW m10, m7, m85412
+ pshufb m10, m95413
+ movu [r2 + r3], m105414
+5415
+ movu m10, [r0 + 2 * r1]5416
+ movu m11, [r0 + 2 * r1 + 8]5417
+ movu m12, [r0 + 2 * r1 + 16]5418
+5419
+ pshufb m13, m10, m55420
+ pshufb m10, m45421
+ pshufb m14, m11, m55422
+ pshufb m11, m45423
+ pshufb m15, m12, m55424
+ pshufb m12, m45425
+5426
+ pmaddwd m10, m05427
+ pmaddwd m13, m15428
+ paddd m10, m135429
+ pmaddwd m13, m14, m35430
+ pmaddwd m16, m11, m25431
+ paddd m13, m165432
+ paddd m10, m135433
+ paddd m10, m65434
+ psrad m10, INTERP_SHIFT_PP5435
+5436
+ pmaddwd m11, m05437
+ pmaddwd m14, m15438
+ paddd m11, m145439
+ pmaddwd m15, m35440
+ pmaddwd m12, m25441
+ paddd m12, m155442
+ paddd m11, m125443
+ paddd m11, m65444
+ psrad m11, INTERP_SHIFT_PP5445
+5446
+ packusdw m10, m115447
+ CLIPW m10, m7, m85448
+ pshufb m10, m95449
+ movu [r2 + 2 * r3], m105450
+5451
+ movu m10, [r0 + r6]5452
+ movu m11, [r0 + r6 + 8]5453
+ movu m12, [r0 + r6 + 16]5454
+5455
+ pshufb m13, m10, m55456
+ pshufb m10, m45457
+ pshufb m14, m11, m55458
+ pshufb m11, m45459
+ pshufb m15, m12, m55460
+ pshufb m12, m45461
+5462
+ pmaddwd m10, m05463
+ pmaddwd m13, m15464
+ paddd m10, m135465
+ pmaddwd m13, m14, m35466
+ pmaddwd m16, m11, m25467
+ paddd m13, m165468
+ paddd m10, m135469
+ paddd m10, m65470
+ psrad m10, INTERP_SHIFT_PP5471
+5472
+ pmaddwd m11, m05473
+ pmaddwd m14, m15474
+ paddd m11, m145475
+ pmaddwd m14, m15, m35476
+ pmaddwd m16, m12, m25477
+ paddd m14, m165478
+ paddd m11, m145479
+ paddd m11, m65480
+ psrad m11, INTERP_SHIFT_PP5481
+5482
+ packusdw m10, m115483
+ CLIPW m10, m7, m85484
+ pshufb m10, m95485
+ movu [r2 + r7], m105486
+5487
+ movu ym10, [r0 + mmsize]5488
+ vinserti32x8 m10, [r0 + r1 + mmsize], 15489
+ movu ym11, [r0 + mmsize + 8]5490
+ vinserti32x8 m11, [r0 + r1 + mmsize + 8], 15491
+ movu ym12, [r0 + mmsize + 16]5492
+ vinserti32x8 m12, [r0 + r1 + mmsize + 16], 15493
+5494
+ pshufb m13, m10, m55495
+ pshufb m10, m45496
+ pshufb m14, m11, m55497
+ pshufb m11, m45498
+ pshufb m15, m12, m55499
+ pshufb m12, m45500
+5501
+ pmaddwd m10, m05502
+ pmaddwd m13, m15503
+ paddd m10, m135504
+ pmaddwd m13, m14, m35505
+ pmaddwd m16, m11, m25506
+ paddd m13, m165507
+ paddd m10, m135508
+ paddd m10, m65509
+ psrad m10, INTERP_SHIFT_PP5510
+5511
+ pmaddwd m11, m05512
+ pmaddwd m14, m15513
+ paddd m11, m145514
+ pmaddwd m15, m35515
+ pmaddwd m12, m25516
+ paddd m12, m155517
+ paddd m11, m125518
+ paddd m11, m65519
+ psrad m11, INTERP_SHIFT_PP5520
+5521
+ packusdw m10, m115522
+ CLIPW m10, m7, m85523
+ pshufb m10, m95524
+ movu [r2 + mmsize], ym105525
+ vextracti32x8 [r2 + r3 + mmsize], m10, 15526
+5527
+ movu ym10, [r0 + 2 * r1 + mmsize]5528
+ vinserti32x8 m10, [r0 + r6 + mmsize], 15529
+ movu ym11, [r0 + 2 * r1 + mmsize + 8]5530
+ vinserti32x8 m11, [r0 + r6 + mmsize + 8], 15531
+ movu ym12, [r0 + 2 * r1 + mmsize + 16]5532
+ vinserti32x8 m12, [r0 + r6 + mmsize + 16], 15533
+5534
+ pshufb m13, m10, m55535
+ pshufb m10, m45536
+ pshufb m14, m11, m55537
+ pshufb m11, m45538
+ pshufb m15, m12, m55539
+ pshufb m12, m45540
+5541
+ pmaddwd m10, m05542
+ pmaddwd m13, m15543
+ paddd m10, m135544
+ pmaddwd m13, m14, m35545
+ pmaddwd m16, m11, m25546
+ paddd m13, m165547
+ paddd m10, m135548
+ paddd m10, m65549
+ psrad m10, INTERP_SHIFT_PP5550
+5551
+ pmaddwd m11, m05552
+ pmaddwd m14, m15553
+ paddd m11, m145554
+ pmaddwd m14, m15, m35555
+ pmaddwd m16, m12, m25556
+ paddd m14, m165557
+ paddd m11, m145558
+ paddd m11, m65559
+ psrad m11, INTERP_SHIFT_PP5560
+5561
+ packusdw m10, m115562
+ CLIPW m10, m7, m85563
+ pshufb m10, m95564
+ movu [r2 + 2 * r3 + mmsize], ym105565
+ vextracti32x8 [r2 + r7 + mmsize], m10, 15566
+%endmacro5567
+5568
+%macro PROCESS_IPFILTER_LUMA_PP_64x2_AVX512 05569
+ ; register map5570
+ ; m0 , m1, m2, m3 - interpolate coeff5571
+ ; m4 , m5 load shuffle order table5572
+ ; m6 - pd_325573
+ ; m7 - zero5574
+ ; m8 - pw_pixel_max5575
+ ; m9 - store shuffle order table5576
+5577
+ movu m10, [r0]5578
+ movu m11, [r0 + 8]5579
+ movu m12, [r0 + 16]5580
+5581
+ pshufb m13, m10, m55582
+ pshufb m10, m45583
+ pshufb m14, m11, m55584
+ pshufb m11, m45585
+ pshufb m15, m12, m55586
+ pshufb m12, m45587
+5588
+ pmaddwd m10, m05589
+ pmaddwd m13, m15590
+ paddd m10, m135591
+ pmaddwd m13, m14, m35592
+ pmaddwd m16, m11, m25593
+ paddd m13, m165594
+ paddd m10, m135595
+ paddd m10, m65596
+ psrad m10, INTERP_SHIFT_PP5597
+5598
+ pmaddwd m11, m05599
+ pmaddwd m14, m15600
+ paddd m11, m145601
+ pmaddwd m15, m35602
+ pmaddwd m12, m25603
+ paddd m12, m155604
+ paddd m11, m125605
+ paddd m11, m65606
+ psrad m11, INTERP_SHIFT_PP5607
+5608
+ packusdw m10, m115609
+ CLIPW m10, m7, m85610
+ pshufb m10, m95611
+ movu [r2], m105612
+5613
+ movu m10, [r0 + mmsize]5614
+ movu m11, [r0 + mmsize + 8]5615
+ movu m12, [r0 + mmsize + 16]5616
+5617
+ pshufb m13, m10, m55618
+ pshufb m10, m45619
+ pshufb m14, m11, m55620
+ pshufb m11, m45621
+ pshufb m15, m12, m55622
+ pshufb m12, m45623
+5624
+ pmaddwd m10, m05625
+ pmaddwd m13, m15626
+ paddd m10, m135627
+ pmaddwd m13, m14, m35628
+ pmaddwd m16, m11, m25629
+ paddd m13, m165630
+ paddd m10, m135631
+ paddd m10, m65632
+ psrad m10, INTERP_SHIFT_PP5633
+5634
+ pmaddwd m11, m05635
+ pmaddwd m14, m15636
+ paddd m11, m145637
+ pmaddwd m15, m35638
+ pmaddwd m12, m25639
+ paddd m12, m155640
+ paddd m11, m125641
+ paddd m11, m65642
+ psrad m11, INTERP_SHIFT_PP5643
+5644
+ packusdw m10, m115645
+ CLIPW m10, m7, m85646
+ pshufb m10, m95647
+ movu [r2 + mmsize], m105648
+5649
+ movu m10, [r0 + r1]5650
+ movu m11, [r0 + r1 + 8]5651
+ movu m12, [r0 + r1 + 16]5652
+5653
+ pshufb m13, m10, m55654
+ pshufb m10, m45655
+ pshufb m14, m11, m55656
+ pshufb m11, m45657
+ pshufb m15, m12, m55658
+ pshufb m12, m45659
+5660
+ pmaddwd m10, m05661
+ pmaddwd m13, m15662
+ paddd m10, m135663
+ pmaddwd m13, m14, m35664
+ pmaddwd m16, m11, m25665
+ paddd m13, m165666
+ paddd m10, m135667
+ paddd m10, m65668
+ psrad m10, INTERP_SHIFT_PP5669
+5670
+ pmaddwd m11, m05671
+ pmaddwd m14, m15672
+ paddd m11, m145673
+ pmaddwd m14, m15, m35674
+ pmaddwd m16, m12, m25675
+ paddd m14, m165676
+ paddd m11, m145677
+ paddd m11, m65678
+ psrad m11, INTERP_SHIFT_PP5679
+5680
+ packusdw m10, m115681
+ CLIPW m10, m7, m85682
+ pshufb m10, m95683
+ movu [r2 + r3], m105684
+5685
+ movu m10, [r0 + r1 + mmsize]5686
+ movu m11, [r0 + r1 + mmsize + 8]5687
+ movu m12, [r0 + r1 + mmsize + 16]5688
+5689
+ pshufb m13, m10, m55690
+ pshufb m10, m45691
+ pshufb m14, m11, m55692
+ pshufb m11, m45693
+ pshufb m15, m12, m55694
+ pshufb m12, m45695
+5696
+ pmaddwd m10, m05697
+ pmaddwd m13, m15698
+ paddd m10, m135699
+ pmaddwd m13, m14, m35700
+ pmaddwd m16, m11, m25701
+ paddd m13, m165702
+ paddd m10, m135703
+ paddd m10, m65704
+ psrad m10, INTERP_SHIFT_PP5705
+5706
+ pmaddwd m11, m05707
+ pmaddwd m14, m15708
+ paddd m11, m145709
+ pmaddwd m14, m15, m35710
+ pmaddwd m16, m12, m25711
+ paddd m14, m165712
+ paddd m11, m145713
+ paddd m11, m65714
+ psrad m11, INTERP_SHIFT_PP5715
+5716
+ packusdw m10, m115717
+ CLIPW m10, m7, m85718
+ pshufb m10, m95719
+ movu [r2 + r3 + mmsize], m105720
+%endmacro5721
+5722
+%macro IPFILTER_LUMA_AVX512_8xN 15723
+INIT_ZMM avx5125724
+cglobal interp_8tap_horiz_pp_8x%1, 5, 8, 175725
+ add r1d, r1d5726
+ add r3d, r3d5727
+ sub r0, 65728
+ mov r4d, r4m5729
+ shl r4d, 45730
+5731
+%ifdef PIC5732
+ lea r5, [tab_LumaCoeff]5733
+ vpbroadcastd m0, [r5 + r4]5734
+ vpbroadcastd m1, [r5 + r4 + 4]5735
+ vpbroadcastd m2, [r5 + r4 + 8]5736
+ vpbroadcastd m3, [r5 + r4 + 12]5737
+%else5738
+ vpbroadcastd m0, [tab_LumaCoeff + r4]5739
+ vpbroadcastd m1, [tab_LumaCoeff + r4 + 4]5740
+ vpbroadcastd m2, [tab_LumaCoeff + r4 + 8]5741
+ vpbroadcastd m3, [tab_LumaCoeff + r4 + 12]5742
+%endif5743
+ vbroadcasti32x8 m4, [interp8_hpp_shuf1_load_avx512]5744
+ vbroadcasti32x8 m5, [interp8_hpp_shuf2_load_avx512]5745
+ vbroadcasti32x8 m6, [pd_32]5746
+ pxor m7, m75747
+ vbroadcasti32x8 m8, [pw_pixel_max]5748
+ vbroadcasti32x8 m9, [interp8_hpp_shuf1_store_avx512]5749
+ lea r6, [3 * r1]5750
+ lea r7, [3 * r3]5751
+5752
+%rep %1/4 - 15753
+ PROCESS_IPFILTER_LUMA_PP_8x4_AVX5125754
+ lea r0, [r0 + 4 * r1]5755
+ lea r2, [r2 + 4 * r3]5756
+%endrep5757
+ PROCESS_IPFILTER_LUMA_PP_8x4_AVX5125758
+ RET5759
+%endmacro5760
+5761
+%if ARCH_X86_645762
+ IPFILTER_LUMA_AVX512_8xN 45763
+ IPFILTER_LUMA_AVX512_8xN 85764
+ IPFILTER_LUMA_AVX512_8xN 165765
+ IPFILTER_LUMA_AVX512_8xN 325766
+%endif5767
+5768
+%macro IPFILTER_LUMA_AVX512_16xN 15769
+INIT_ZMM avx5125770
+cglobal interp_8tap_horiz_pp_16x%1, 5,8,175771
+ add r1d, r1d5772
+ add r3d, r3d5773
+ sub r0, 65774
+ mov r4d, r4m5775
+ shl r4d, 45776
+5777
+%ifdef PIC5778
+ lea r5, [tab_LumaCoeff]5779
+ vpbroadcastd m0, [r5 + r4]5780
+ vpbroadcastd m1, [r5 + r4 + 4]5781
+ vpbroadcastd m2, [r5 + r4 + 8]5782
+ vpbroadcastd m3, [r5 + r4 + 12]5783
+%else5784
+ vpbroadcastd m0, [tab_LumaCoeff + r4]5785
+ vpbroadcastd m1, [tab_LumaCoeff + r4 + 4]5786
+ vpbroadcastd m2, [tab_LumaCoeff + r4 + 8]5787
+ vpbroadcastd m3, [tab_LumaCoeff + r4 + 12]5788
+%endif5789
+ vbroadcasti32x8 m4, [interp8_hpp_shuf1_load_avx512]5790
+ vbroadcasti32x8 m5, [interp8_hpp_shuf2_load_avx512]5791
+ vbroadcasti32x8 m6, [pd_32]5792
+ pxor m7, m75793
+ vbroadcasti32x8 m8, [pw_pixel_max]5794
+ vbroadcasti32x8 m9, [interp8_hpp_shuf1_store_avx512]5795
+ lea r6, [3 * r1]5796
+ lea r7, [3 * r3]5797
+5798
+%rep %1/4 - 15799
+ PROCESS_IPFILTER_LUMA_PP_16x4_AVX5125800
+ lea r0, [r0 + 4 * r1]5801
+ lea r2, [r2 + 4 * r3]5802
+%endrep5803
+ PROCESS_IPFILTER_LUMA_PP_16x4_AVX5125804
+ RET5805
+%endmacro5806
+5807
+%if ARCH_X86_645808
+IPFILTER_LUMA_AVX512_16xN 45809
+IPFILTER_LUMA_AVX512_16xN 85810
+IPFILTER_LUMA_AVX512_16xN 125811
+IPFILTER_LUMA_AVX512_16xN 165812
+IPFILTER_LUMA_AVX512_16xN 325813
+IPFILTER_LUMA_AVX512_16xN 645814
+%endif5815
+5816
+%if ARCH_X86_645817
+INIT_ZMM avx5125818
+cglobal interp_8tap_horiz_pp_24x32, 5, 8, 175819
+ add r1d, r1d5820
+ add r3d, r3d5821
+ sub r0, 65822
+ mov r4d, r4m5823
+ shl r4d, 45824
+5825
+%ifdef PIC5826
+ lea r5, [tab_LumaCoeff]5827
+ vpbroadcastd m0, [r5 + r4]5828
+ vpbroadcastd m1, [r5 + r4 + 4]5829
+ vpbroadcastd m2, [r5 + r4 + 8]5830
+ vpbroadcastd m3, [r5 + r4 + 12]5831
+%else5832
+ vpbroadcastd m0, [tab_LumaCoeff + r4]5833
+ vpbroadcastd m1, [tab_LumaCoeff + r4 + 4]5834
+ vpbroadcastd m2, [tab_LumaCoeff + r4 + 8]5835
+ vpbroadcastd m3, [tab_LumaCoeff + r4 + 12]5836
+%endif5837
+ vbroadcasti32x8 m4, [interp8_hpp_shuf1_load_avx512]5838
+ vbroadcasti32x8 m5, [interp8_hpp_shuf2_load_avx512]5839
+ vbroadcasti32x8 m6, [pd_32]5840
+ pxor m7, m75841
+ vbroadcasti32x8 m8, [pw_pixel_max]5842
+ vbroadcasti32x8 m9, [interp8_hpp_shuf1_store_avx512]5843
+ lea r6, [3 * r1]5844
+ lea r7, [3 * r3]5845
+5846
+%rep 75847
+ PROCESS_IPFILTER_LUMA_PP_24x4_AVX5125848
+ lea r0, [r0 + 4 * r1]5849
+ lea r2, [r2 + 4 * r3]5850
+%endrep5851
+ PROCESS_IPFILTER_LUMA_PP_24x4_AVX5125852
+ RET5853
+%endif5854
+5855
+%macro IPFILTER_LUMA_AVX512_32xN 15856
+INIT_ZMM avx5125857
+cglobal interp_8tap_horiz_pp_32x%1, 5,6,175858
+ add r1d, r1d5859
+ add r3d, r3d5860
+ sub r0, 65861
+ mov r4d, r4m5862
+ shl r4d, 45863
+5864
+%ifdef PIC5865
+ lea r5, [tab_LumaCoeff]5866
+ vpbroadcastd m0, [r5 + r4]5867
+ vpbroadcastd m1, [r5 + r4 + 4]5868
+ vpbroadcastd m2, [r5 + r4 + 8]5869
+ vpbroadcastd m3, [r5 + r4 + 12]5870
+%else5871
+ vpbroadcastd m0, [tab_LumaCoeff + r4]5872
+ vpbroadcastd m1, [tab_LumaCoeff + r4 + 4]5873
+ vpbroadcastd m2, [tab_LumaCoeff + r4 + 8]5874
+ vpbroadcastd m3, [tab_LumaCoeff + r4 + 12]5875
+%endif5876
+ vbroadcasti32x8 m4, [interp8_hpp_shuf1_load_avx512]5877
+ vbroadcasti32x8 m5, [interp8_hpp_shuf2_load_avx512]5878
+ vbroadcasti32x8 m6, [pd_32]5879
+ pxor m7, m75880
+ vbroadcasti32x8 m8, [pw_pixel_max]5881
+ vbroadcasti32x8 m9, [interp8_hpp_shuf1_store_avx512]5882
+5883
+%rep %1/2 - 15884
+ PROCESS_IPFILTER_LUMA_PP_32x2_AVX5125885
+ lea r0, [r0 + 2 * r1]5886
+ lea r2, [r2 + 2 * r3]5887
+%endrep5888
+ PROCESS_IPFILTER_LUMA_PP_32x2_AVX5125889
+ RET5890
+%endmacro5891
+5892
+%if ARCH_X86_645893
+IPFILTER_LUMA_AVX512_32xN 85894
+IPFILTER_LUMA_AVX512_32xN 165895
+IPFILTER_LUMA_AVX512_32xN 245896
+IPFILTER_LUMA_AVX512_32xN 325897
+IPFILTER_LUMA_AVX512_32xN 645898
+%endif5899
+5900
+%macro IPFILTER_LUMA_AVX512_64xN 15901
+INIT_ZMM avx5125902
+cglobal interp_8tap_horiz_pp_64x%1, 5,6,175903
+ add r1d, r1d5904
+ add r3d, r3d5905
+ sub r0, 65906
+ mov r4d, r4m5907
+ shl r4d, 45908
+5909
+%ifdef PIC5910
+ lea r5, [tab_LumaCoeff]5911
+ vpbroadcastd m0, [r5 + r4]5912
+ vpbroadcastd m1, [r5 + r4 + 4]5913
+ vpbroadcastd m2, [r5 + r4 + 8]5914
+ vpbroadcastd m3, [r5 + r4 + 12]5915
+%else5916
+ vpbroadcastd m0, [tab_LumaCoeff + r4]5917
+ vpbroadcastd m1, [tab_LumaCoeff + r4 + 4]5918
+ vpbroadcastd m2, [tab_LumaCoeff + r4 + 8]5919
+ vpbroadcastd m3, [tab_LumaCoeff + r4 + 12]5920
+%endif5921
+ vbroadcasti32x8 m4, [interp8_hpp_shuf1_load_avx512]5922
+ vbroadcasti32x8 m5, [interp8_hpp_shuf2_load_avx512]5923
+ vbroadcasti32x8 m6, [pd_32]5924
+ pxor m7, m75925
+ vbroadcasti32x8 m8, [pw_pixel_max]5926
+ vbroadcasti32x8 m9, [interp8_hpp_shuf1_store_avx512]5927
+5928
+%rep %1/2 - 15929
+ PROCESS_IPFILTER_LUMA_PP_64x2_AVX5125930
+ lea r0, [r0 + 2 * r1]5931
+ lea r2, [r2 + 2 * r3]5932
+%endrep5933
+ PROCESS_IPFILTER_LUMA_PP_64x2_AVX5125934
+ RET5935
+%endmacro5936
+5937
+%if ARCH_X86_645938
+IPFILTER_LUMA_AVX512_64xN 165939
+IPFILTER_LUMA_AVX512_64xN 325940
+IPFILTER_LUMA_AVX512_64xN 485941
+IPFILTER_LUMA_AVX512_64xN 645942
+%endif5943
+5944
+%if ARCH_X86_645945
+INIT_ZMM avx5125946
+cglobal interp_8tap_horiz_pp_48x64, 5,8,175947
+ add r1d, r1d5948
+ add r3d, r3d5949
+ sub r0, 65950
+ mov r4d, r4m5951
+ shl r4d, 45952
+5953
+%ifdef PIC5954
+ lea r5, [tab_LumaCoeff]5955
+ vpbroadcastd m0, [r5 + r4]5956
+ vpbroadcastd m1, [r5 + r4 + 4]5957
+ vpbroadcastd m2, [r5 + r4 + 8]5958
+ vpbroadcastd m3, [r5 + r4 + 12]5959
+%else5960
+ vpbroadcastd m0, [tab_LumaCoeff + r4]5961
+ vpbroadcastd m1, [tab_LumaCoeff + r4 + 4]5962
+ vpbroadcastd m2, [tab_LumaCoeff + r4 + 8]5963
+ vpbroadcastd m3, [tab_LumaCoeff + r4 + 12]5964
+%endif5965
+ vbroadcasti32x8 m4, [interp8_hpp_shuf1_load_avx512]5966
+ vbroadcasti32x8 m5, [interp8_hpp_shuf2_load_avx512]5967
+ vbroadcasti32x8 m6, [pd_32]5968
+ pxor m7, m75969
+ vbroadcasti32x8 m8, [pw_pixel_max]5970
+ vbroadcasti32x8 m9, [interp8_hpp_shuf1_store_avx512]5971
+ lea r6, [3 * r1]5972
+ lea r7, [3 * r3]5973
+5974
+%rep 155975
+ PROCESS_IPFILTER_LUMA_PP_48x4_AVX5125976
+ lea r0, [r0 + 4 * r1]5977
+ lea r2, [r2 + 4 * r3]5978
+%endrep5979
+ PROCESS_IPFILTER_LUMA_PP_48x4_AVX5125980
+ RET5981
+%endif5982
+;-------------------------------------------------------------------------------------------------------------5983
+;avx512 luma_hps code start5984
+;-------------------------------------------------------------------------------------------------------------5985
+5986
+%macro PROCESS_IPFILTER_LUMA_PS_32x2_AVX512 05987
+ ; register map5988
+ ; m0, m1, m2, m3 - interpolate coeff5989
+ ; m4, m5 - shuffle load order table5990
+ ; m6 - INTERP_OFFSET_PS5991
+ ; m7 - shuffle store order table5992
+5993
+ movu m8, [r0]5994
+ movu m9, [r0 + 8]5995
+ movu m10, [r0 + 16]5996
+5997
+ pshufb m11, m8, m55998
+ pshufb m8, m45999
+ pmaddwd m8, m06000
+ pmaddwd m11, m16001
+ paddd m8, m116002
+ pshufb m12, m9, m56003
+ pshufb m9, m46004
+ pmaddwd m11, m12, m36005
+ pmaddwd m14, m9, m26006
+ paddd m11, m146007
+6008
+ paddd m8, m116009
+ paddd m8, m66010
+ psrad m8, INTERP_SHIFT_PS6011
+6012
+ pshufb m13, m10, m56013
+ pshufb m10, m46014
+ pmaddwd m9, m06015
+ pmaddwd m12, m16016
+ paddd m9, m126017
+ pmaddwd m13, m36018
+ pmaddwd m10, m26019
+ paddd m10, m136020
+6021
+ paddd m9, m106022
+ paddd m9, m66023
+ psrad m9, INTERP_SHIFT_PS6024
+6025
+ packssdw m8, m96026
+ pshufb m8, m76027
+ movu [r2], m86028
+6029
+ movu m8, [r0 + r1]6030
+ movu m9, [r0 + r1 + 8]6031
+ movu m10, [r0 + r1 + 16]6032
+6033
+ pshufb m11, m8, m56034
+ pshufb m8, m46035
+ pmaddwd m8, m06036
+ pmaddwd m11, m16037
+ paddd m8, m116038
+ pshufb m12, m9, m56039
+ pshufb m9, m46040
+ pmaddwd m11, m12, m36041
+ pmaddwd m14, m9, m26042
+ paddd m11, m146043
+6044
+ paddd m8, m116045
+ paddd m8, m66046
+ psrad m8, INTERP_SHIFT_PS6047
+6048
+ pshufb m13, m10, m56049
+ pshufb m10, m46050
+ pmaddwd m9, m06051
+ pmaddwd m12, m16052
+ paddd m9, m126053
+ pmaddwd m12, m13, m36054
+ pmaddwd m14, m10, m26055
+ paddd m12, m146056
+6057
+ paddd m9, m126058
+ paddd m9, m66059
+ psrad m9, INTERP_SHIFT_PS6060
+6061
+ packssdw m8, m96062
+ pshufb m8, m76063
+ movu [r2 + r3],m86064
+%endmacro6065
+6066
+%macro PROCESS_IPFILTER_LUMA_PS_32x1_AVX512 06067
+ movu m8, [r0]6068
+ movu m9, [r0 + 8]6069
+ movu m10, [r0 + 16]6070
+6071
+ pshufb m11, m8, m56072
+ pshufb m8, m46073
+ pmaddwd m8, m06074
+ pmaddwd m11, m16075
+ paddd m8, m116076
+ pshufb m12, m9, m56077
+ pshufb m9, m46078
+ pmaddwd m11, m12, m36079
+ pmaddwd m14, m9, m26080
+ paddd m11, m146081
+6082
+ paddd m8, m116083
+ paddd m8, m66084
+ psrad m8, INTERP_SHIFT_PS6085
+6086
+ pshufb m13, m10, m56087
+ pshufb m10, m46088
+ pmaddwd m9, m06089
+ pmaddwd m12, m16090
+ paddd m9, m126091
+ pmaddwd m13, m36092
+ pmaddwd m10, m26093
+ paddd m10, m136094
+6095
+ paddd m9, m106096
+ paddd m9, m66097
+ psrad m9, INTERP_SHIFT_PS6098
+6099
+ packssdw m8, m96100
+ pshufb m8, m76101
+ movu [r2], m86102
+%endmacro6103
+6104
+%macro IPFILTER_LUMA_PS_AVX512_32xN 16105
+INIT_ZMM avx5126106
+cglobal interp_8tap_horiz_ps_32x%1, 4,7,156107
+ shl r1d, 16108
+ shl r3d, 16109
+ mov r4d, r4m6110
+ mov r5d, r5m6111
+ shl r4d, 66112
+6113
+%ifdef PIC6114
+ lea r6, [tab_LumaCoeffH_avx512]6115
+ vpbroadcastd m0, [r6 + r4]6116
+ vpbroadcastd m1, [r6 + r4 + 4]6117
+ vpbroadcastd m2, [r6 + r4 + 8]6118
+ vpbroadcastd m3, [r6 + r4 + 12]6119
+%else6120
+ vpbroadcastd m0, [tab_LumaCoeffH_avx512 + r4]6121
+ vpbroadcastd m1, [tab_LumaCoeffH_avx512 + r4 + 4]6122
+ vpbroadcastd m2, [tab_LumaCoeffH_avx512 + r4 + 8]6123
+ vpbroadcastd m3, [tab_LumaCoeffH_avx512 + r4 + 12]6124
+%endif6125
+ vbroadcasti32x8 m4, [interp8_hpp_shuf1_load_avx512]6126
+ vbroadcasti32x8 m5, [interp8_hpp_shuf2_load_avx512]6127
+ vbroadcasti32x4 m6, [INTERP_OFFSET_PS]6128
+ vbroadcasti32x8 m7, [interp8_hpp_shuf1_store_avx512]6129
+6130
+ sub r0, 66131
+ mov r4d, %16132
+ test r5d, r5d6133
+ jz .loop6134
+ lea r6, [r1 * 3]6135
+ sub r0, r66136
+ add r4d, 76137
+ PROCESS_IPFILTER_LUMA_PS_32x1_AVX5126138
+ lea r0, [r0 + r1]6139
+ lea r2, [r2 + r3]6140
+ dec r4d6141
+6142
+.loop:6143
+ PROCESS_IPFILTER_LUMA_PS_32x2_AVX5126144
+ lea r0, [r0 + 2 * r1]6145
+ lea r2, [r2 + 2 * r3]6146
+ sub r4d, 26147
+ jnz .loop6148
+ RET6149
+%endmacro6150
+6151
+%if ARCH_X86_646152
+IPFILTER_LUMA_PS_AVX512_32xN 86153
+IPFILTER_LUMA_PS_AVX512_32xN 166154
+IPFILTER_LUMA_PS_AVX512_32xN 246155
+IPFILTER_LUMA_PS_AVX512_32xN 326156
+IPFILTER_LUMA_PS_AVX512_32xN 646157
+%endif6158
+6159
+%macro PROCESS_IPFILTER_LUMA_PS_64x2_AVX512 06160
+ ; register map6161
+ ; m0, m1, m2, m3 - interpolate coeff6162
+ ; m4, m5 - shuffle load order table6163
+ ; m6 - INTERP_OFFSET_PS6164
+ ; m7 - shuffle store order table6165
+6166
+ movu m8, [r0]6167
+ movu m9, [r0 + 8]6168
+ movu m10, [r0 + 16]6169
+6170
+ pshufb m11, m8, m56171
+ pshufb m8, m46172
+ pmaddwd m8, m06173
+ pmaddwd m11, m16174
+ paddd m8, m116175
+ pshufb m12, m9, m56176
+ pshufb m9, m46177
+ pmaddwd m11, m12, m36178
+ pmaddwd m14, m9, m26179
+ paddd m11, m146180
+6181
+ paddd m8, m116182
+ paddd m8, m66183
+ psrad m8, INTERP_SHIFT_PS6184
+6185
+ pshufb m13, m10, m56186
+ pshufb m10, m46187
+ pmaddwd m9, m06188
+ pmaddwd m12, m16189
+ paddd m9, m126190
+ pmaddwd m13, m36191
+ pmaddwd m10, m26192
+ paddd m10, m136193
+6194
+ paddd m9, m106195
+ paddd m9, m66196
+ psrad m9, INTERP_SHIFT_PS6197
+6198
+ packssdw m8, m96199
+ pshufb m8, m76200
+ movu [r2], m86201
+6202
+ movu m8, [r0 + mmsize]6203
+ movu m9, [r0 + mmsize + 8]6204
+ movu m10, [r0 + mmsize + 16]6205
+6206
+ pshufb m11, m8, m56207
+ pshufb m8, m46208
+ pmaddwd m8, m06209
+ pmaddwd m11, m16210
+ paddd m8, m116211
+ pshufb m12, m9, m56212
+ pshufb m9, m46213
+ pmaddwd m11, m12, m36214
+ pmaddwd m14, m9, m26215
+ paddd m11, m146216
+ paddd m8, m116217
+ paddd m8, m66218
+ psrad m8, INTERP_SHIFT_PS6219
+6220
+ pshufb m13, m10, m56221
+ pshufb m10, m46222
+ pmaddwd m9, m06223
+ pmaddwd m12, m16224
+ paddd m9, m126225
+ pmaddwd m13, m36226
+ pmaddwd m10, m26227
+ paddd m10, m136228
+ paddd m9, m106229
+ paddd m9, m66230
+ psrad m9, INTERP_SHIFT_PS6231
+6232
+ packssdw m8, m96233
+ pshufb m8, m76234
+ movu [r2 + mmsize], m86235
+6236
+ movu m8, [r0 + r1]6237
+ movu m9, [r0 + r1 + 8]6238
+ movu m10, [r0 + r1 + 16]6239
+6240
+ pshufb m11, m8, m56241
+ pshufb m8, m46242
+ pmaddwd m8, m06243
+ pmaddwd m11, m16244
+ paddd m8, m116245
+ pshufb m12, m9, m56246
+ pshufb m9, m46247
+ pmaddwd m11, m12, m36248
+ pmaddwd m14, m9, m26249
+ paddd m11, m146250
+ paddd m8, m116251
+ paddd m8, m66252
+ psrad m8, INTERP_SHIFT_PS6253
+6254
+ pshufb m13, m10, m56255
+ pshufb m10, m46256
+ pmaddwd m9, m06257
+ pmaddwd m12, m16258
+ paddd m9, m126259
+ pmaddwd m12, m13, m36260
+ pmaddwd m14, m10, m26261
+ paddd m12, m146262
+ paddd m9, m126263
+ paddd m9, m66264
+ psrad m9, INTERP_SHIFT_PS6265
+6266
+ packssdw m8, m96267
+ pshufb m8, m76268
+ movu [r2 + r3],m86269
+6270
+ movu m8, [r0 + r1 + mmsize]6271
+ movu m9, [r0 + r1 + mmsize + 8]6272
+ movu m10, [r0 + r1 + mmsize + 16]6273
+6274
+ pshufb m11, m8, m56275
+ pshufb m8, m46276
+ pmaddwd m8, m06277
+ pmaddwd m11, m16278
+ paddd m8, m116279
+ pshufb m12, m9, m56280
+ pshufb m9, m46281
+ pmaddwd m11, m12, m36282
+ pmaddwd m14, m9, m26283
+ paddd m11, m146284
+ paddd m8, m116285
+ paddd m8, m66286
+ psrad m8, INTERP_SHIFT_PS6287
+6288
+ pshufb m13, m10, m56289
+ pshufb m10, m46290
+ pmaddwd m9, m06291
+ pmaddwd m12, m16292
+ paddd m9, m126293
+ pmaddwd m12, m13, m36294
+ pmaddwd m14, m10, m26295
+ paddd m12, m146296
+ paddd m9, m126297
+ paddd m9, m66298
+ psrad m9, INTERP_SHIFT_PS6299
+6300
+ packssdw m8, m96301
+ pshufb m8, m76302
+ movu [r2 + r3 + mmsize], m86303
+%endmacro6304
+6305
+%macro PROCESS_IPFILTER_LUMA_PS_64x1_AVX512 06306
+6307
+ movu m8, [r0]6308
+ movu m9, [r0 + 8]6309
+ movu m10, [r0 + 16]6310
+6311
+ pshufb m11, m8, m56312
+ pshufb m8, m46313
+ pmaddwd m8, m06314
+ pmaddwd m11, m16315
+ paddd m8, m116316
+ pshufb m12, m9, m56317
+ pshufb m9, m46318
+ pmaddwd m11, m12, m36319
+ pmaddwd m14, m9, m26320
+ paddd m11, m146321
+ paddd m8, m116322
+ paddd m8, m66323
+ psrad m8, INTERP_SHIFT_PS6324
+6325
+ pshufb m13, m10, m56326
+ pshufb m10, m46327
+ pmaddwd m9, m06328
+ pmaddwd m12, m16329
+ paddd m9, m126330
+ pmaddwd m13, m36331
+ pmaddwd m10, m26332
+ paddd m10, m136333
+ paddd m9, m106334
+ paddd m9, m66335
+ psrad m9, INTERP_SHIFT_PS6336
+6337
+ packssdw m8, m96338
+ pshufb m8, m76339
+ movu [r2], m86340
+6341
+ movu m8, [r0 + mmsize]6342
+ movu m9, [r0 + mmsize + 8]6343
+ movu m10, [r0 + mmsize + 16]6344
+6345
+ pshufb m11, m8, m56346
+ pshufb m8, m46347
+ pmaddwd m8, m06348
+ pmaddwd m11, m16349
+ paddd m8, m116350
+ pshufb m12, m9, m56351
+ pshufb m9, m46352
+ pmaddwd m11, m12, m36353
+ pmaddwd m14, m9, m26354
+ paddd m11, m146355
+ paddd m8, m116356
+ paddd m8, m66357
+ psrad m8, INTERP_SHIFT_PS6358
+6359
+ pshufb m13, m10, m56360
+ pshufb m10, m46361
+ pmaddwd m9, m06362
+ pmaddwd m12, m16363
+ paddd m9, m126364
+ pmaddwd m13, m36365
+ pmaddwd m10, m26366
+ paddd m10, m136367
+ paddd m9, m106368
+ paddd m9, m66369
+ psrad m9, INTERP_SHIFT_PS6370
+6371
+ packssdw m8, m96372
+ pshufb m8, m76373
+ movu [r2 + mmsize], m86374
+%endmacro6375
+6376
+%macro IPFILTER_LUMA_PS_AVX512_64xN 16377
+INIT_ZMM avx5126378
+cglobal interp_8tap_horiz_ps_64x%1, 4,7,156379
+ shl r1d, 16380
+ shl r3d, 16381
+ mov r4d, r4m6382
+ mov r5d, r5m6383
+ shl r4d, 66384
+6385
+%ifdef PIC6386
+ lea r6, [tab_LumaCoeffH_avx512]6387
+ vpbroadcastd m0, [r6 + r4]6388
+ vpbroadcastd m1, [r6 + r4 + 4]6389
+ vpbroadcastd m2, [r6 + r4 + 8]6390
+ vpbroadcastd m3, [r6 + r4 + 12]6391
+%else6392
+ vpbroadcastd m0, [tab_LumaCoeffH_avx512 + r4]6393
+ vpbroadcastd m1, [tab_LumaCoeffH_avx512 + r4 + 4]6394
+ vpbroadcastd m2, [tab_LumaCoeffH_avx512 + r4 + 8]6395
+ vpbroadcastd m3, [tab_LumaCoeffH_avx512 + r4 + 12]6396
+%endif6397
+ vbroadcasti32x8 m4, [interp8_hpp_shuf1_load_avx512]6398
+ vbroadcasti32x8 m5, [interp8_hpp_shuf2_load_avx512]6399
+ vbroadcasti32x4 m6, [INTERP_OFFSET_PS]6400
+ vbroadcasti32x8 m7, [interp8_hpp_shuf1_store_avx512]6401
+6402
+ sub r0, 66403
+ mov r4d, %16404
+ test r5d, r5d6405
+ jz .loop6406
+ lea r6, [r1 * 3]6407
+ sub r0, r66408
+ add r4d, 76409
+ PROCESS_IPFILTER_LUMA_PS_64x1_AVX5126410
+ lea r0, [r0 + r1]6411
+ lea r2, [r2 + r3]6412
+ dec r4d6413
+6414
+.loop:6415
+ PROCESS_IPFILTER_LUMA_PS_64x2_AVX5126416
+ lea r0, [r0 + 2 * r1]6417
+ lea r2, [r2 + 2 * r3]6418
+ sub r4d, 26419
+ jnz .loop6420
+ RET6421
+%endmacro6422
+6423
+%if ARCH_X86_646424
+IPFILTER_LUMA_PS_AVX512_64xN 166425
+IPFILTER_LUMA_PS_AVX512_64xN 326426
+IPFILTER_LUMA_PS_AVX512_64xN 486427
+IPFILTER_LUMA_PS_AVX512_64xN 646428
+%endif6429
+6430
+%macro PROCESS_IPFILTER_LUMA_PS_16x4_AVX512 06431
+ ; register map6432
+ ; m0, m1, m2, m3 - interpolate coeff6433
+ ; m4, m5 - shuffle load order table6434
+ ; m6 - INTERP_OFFSET_PS6435
+ ; m7 - shuffle store order table6436
+6437
+ movu ym8, [r0]6438
+ vinserti32x8 m8, [r0 + r1], 16439
+ movu ym9, [r0 + 8]6440
+ vinserti32x8 m9, [r0 + r1 + 8], 16441
+ movu ym10, [r0 + 16]6442
+ vinserti32x8 m10, [r0 + r1 + 16], 16443
+6444
+ pshufb m11, m8, m56445
+ pshufb m8, m46446
+ pmaddwd m8, m06447
+ pmaddwd m11, m16448
+ paddd m8, m116449
+ pshufb m12, m9, m56450
+ pshufb m9, m46451
+ pmaddwd m11, m12, m36452
+ pmaddwd m14, m9, m26453
+ paddd m11, m146454
+ paddd m8, m116455
+ paddd m8, m66456
+ psrad m8, INTERP_SHIFT_PS6457
+6458
+ pshufb m13, m10, m56459
+ pshufb m10, m46460
+ pmaddwd m9, m06461
+ pmaddwd m12, m16462
+ paddd m9, m126463
+ pmaddwd m13, m36464
+ pmaddwd m10, m26465
+ paddd m10, m136466
+ paddd m9, m106467
+ paddd m9, m66468
+ psrad m9, INTERP_SHIFT_PS6469
+6470
+ packssdw m8, m96471
+ pshufb m8, m76472
+ movu [r2], ym86473
+ vextracti32x8 [r2 + r3],m8, 16474
+6475
+ movu ym8, [r0 + 2 * r1]6476
+ vinserti32x8 m8, [r0 + r6], 16477
+ movu ym9, [r0 + 2 * r1 + 8]6478
+ vinserti32x8 m9, [r0 + r6 + 8], 16479
+ movu ym10, [r0 + 2 * r1 + 16]6480
+ vinserti32x8 m10, [r0 + r6 + 16], 16481
+6482
+ pshufb m11, m8, m56483
+ pshufb m8, m46484
+ pmaddwd m8, m06485
+ pmaddwd m11, m16486
+ paddd m8, m116487
+ pshufb m12, m9, m56488
+ pshufb m9, m46489
+ pmaddwd m11, m12, m36490
+ pmaddwd m14, m9, m26491
+ paddd m11, m146492
+ paddd m8, m116493
+ paddd m8, m66494
+ psrad m8, INTERP_SHIFT_PS6495
+6496
+ pshufb m13, m10, m56497
+ pshufb m10, m46498
+ pmaddwd m9, m06499
+ pmaddwd m12, m16500
+ paddd m9, m126501
+ pmaddwd m12, m13, m36502
+ pmaddwd m14, m10, m26503
+ paddd m12, m146504
+ paddd m9, m126505
+ paddd m9, m66506
+ psrad m9, INTERP_SHIFT_PS6507
+6508
+ packssdw m8, m96509
+ pshufb m8, m76510
+ movu [r2 + 2 * r3], ym86511
+ vextracti32x8 [r2 + r7], m8, 16512
+%endmacro6513
+6514
+%macro PROCESS_IPFILTER_LUMA_PS_16x3_AVX512 06515
+ movu ym8, [r0]6516
+ vinserti32x8 m8, [r0 + r1], 16517
+ movu ym9, [r0 + 8]6518
+ vinserti32x8 m9, [r0 + r1 + 8], 16519
+ movu ym10, [r0 + 16]6520
+ vinserti32x8 m10, [r0 + r1 + 16], 16521
+6522
+ pshufb m11, m8, m56523
+ pshufb m8, m46524
+ pmaddwd m8, m06525
+ pmaddwd m11, m16526
+ paddd m8, m116527
+ pshufb m12, m9, m56528
+ pshufb m9, m46529
+ pmaddwd m11, m12, m36530
+ pmaddwd m14, m9, m26531
+ paddd m11, m146532
+ paddd m8, m116533
+ paddd m8, m66534
+ psrad m8, INTERP_SHIFT_PS6535
+6536
+ pshufb m13, m10, m56537
+ pshufb m10, m46538
+ pmaddwd m9, m06539
+ pmaddwd m12, m16540
+ paddd m9, m126541
+ pmaddwd m13, m36542
+ pmaddwd m10, m26543
+ paddd m10, m136544
+ paddd m9, m106545
+ paddd m9, m66546
+ psrad m9, INTERP_SHIFT_PS6547
+6548
+ packssdw m8, m96549
+ pshufb m8, m76550
+ movu [r2], ym86551
+ vextracti32x8 [r2 + r3],m8, 16552
+6553
+ movu ym8, [r0 + 2 * r1]6554
+ movu ym9, [r0 + 2 * r1 + 8]6555
+ movu ym10, [r0 + 2 * r1 + 16]6556
+6557
+ pshufb ym11, ym8, ym56558
+ pshufb ym8, ym46559
+ pmaddwd ym8, ym06560
+ pmaddwd ym11, ym16561
+ paddd ym8, ym116562
+ pshufb ym12, ym9, ym56563
+ pshufb ym9, ym46564
+ pmaddwd ym11, ym12, ym36565
+ pmaddwd ym14, ym9, ym26566
+ paddd ym11, ym146567
+ paddd ym8, ym116568
+ paddd ym8, ym66569
+ psrad ym8, INTERP_SHIFT_PS6570
+6571
+ pshufb ym13, ym10, ym56572
+ pshufb ym10, ym46573
+ pmaddwd ym9, ym06574
+ pmaddwd ym12, ym16575
+ paddd ym9, ym126576
+ pmaddwd ym12, ym13, ym36577
+ pmaddwd ym14, ym10, ym26578
+ paddd ym12, ym146579
+ paddd ym9, ym126580
+ paddd ym9, ym66581
+ psrad ym9, INTERP_SHIFT_PS6582
+6583
+ packssdw ym8, ym96584
+ pshufb ym8, ym76585
+ movu [r2 + 2 * r3], ym86586
+%endmacro6587
+6588
+6589
+%macro IPFILTER_LUMA_PS_AVX512_16xN 16590
+INIT_ZMM avx5126591
+cglobal interp_8tap_horiz_ps_16x%1, 4,9,156592
+ shl r1d, 16593
+ shl r3d, 16594
+ mov r4d, r4m6595
+ mov r5d, r5m6596
+ shl r4d, 66597
+6598
+ lea r6, [3 * r1]6599
+ lea r7, [3 * r3] 6600
+%ifdef PIC6601
+ lea r8, [tab_LumaCoeffH_avx512]6602
+ vpbroadcastd m0, [r8 + r4]6603
+ vpbroadcastd m1, [r8 + r4 + 4]6604
+ vpbroadcastd m2, [r8 + r4 + 8]6605
+ vpbroadcastd m3, [r8 + r4 + 12]6606
+%else6607
+ vpbroadcastd m0, [tab_LumaCoeffH_avx512 + r4]6608
+ vpbroadcastd m1, [tab_LumaCoeffH_avx512 + r4 + 4]6609
+ vpbroadcastd m2, [tab_LumaCoeffH_avx512 + r4 + 8]6610
+ vpbroadcastd m3, [tab_LumaCoeffH_avx512 + r4 + 12]6611
+%endif6612
+ vbroadcasti32x8 m4, [interp8_hpp_shuf1_load_avx512]6613
+ vbroadcasti32x8 m5, [interp8_hpp_shuf2_load_avx512]6614
+ vbroadcasti32x4 m6, [INTERP_OFFSET_PS]6615
+ vbroadcasti32x8 m7, [interp8_hpp_shuf1_store_avx512]6616
+6617
+ sub r0, 66618
+ mov r4d, %16619
+ test r5d, r5d6620
+ jz .loop6621
+ lea r6, [r1 * 3]6622
+ sub r0, r66623
+ add r4d, 76624
+ PROCESS_IPFILTER_LUMA_PS_16x3_AVX5126625
+ lea r0, [r0 + r6]6626
+ lea r2, [r2 + r7]6627
+ sub r4d, 36628
+6629
+.loop:6630
+ PROCESS_IPFILTER_LUMA_PS_16x4_AVX5126631
+ lea r0, [r0 + 4 * r1]6632
+ lea r2, [r2 + 4 * r3]6633
+ sub r4d, 46634
+ jnz .loop6635
+ RET6636
+%endmacro6637
+6638
+%if ARCH_X86_646639
+IPFILTER_LUMA_PS_AVX512_16xN 46640
+IPFILTER_LUMA_PS_AVX512_16xN 86641
+IPFILTER_LUMA_PS_AVX512_16xN 126642
+IPFILTER_LUMA_PS_AVX512_16xN 166643
+IPFILTER_LUMA_PS_AVX512_16xN 326644
+IPFILTER_LUMA_PS_AVX512_16xN 646645
+%endif6646
+6647
+%macro PROCESS_IPFILTER_LUMA_PS_48x4_AVX512 06648
+ ; register map6649
+ ; m0, m1, m2, m3 - interpolate coeff6650
+ ; m4, m5 - shuffle load order table6651
+ ; m6 - INTERP_OFFSET_PS6652
+ ; m7 - shuffle store order table6653
+6654
+ movu m8, [r0]6655
+ movu m9, [r0 + 8]6656
+ movu m10, [r0 + 16]6657
+6658
+ pshufb m11, m8, m56659
+ pshufb m8, m46660
+ pmaddwd m8, m06661
+ pmaddwd m11, m16662
+ paddd m8, m116663
+ pshufb m12, m9, m56664
+ pshufb m9, m46665
+ pmaddwd m11, m12, m36666
+ pmaddwd m14, m9, m26667
+ paddd m11, m146668
+ paddd m8, m116669
+ paddd m8, m66670
+ psrad m8, INTERP_SHIFT_PS6671
+6672
+ pshufb m13, m10, m56673
+ pshufb m10, m46674
+ pmaddwd m9, m06675
+ pmaddwd m12, m16676
+ paddd m9, m126677
+ pmaddwd m13, m36678
+ pmaddwd m10, m26679
+ paddd m10, m136680
+ paddd m9, m106681
+ paddd m9, m66682
+ psrad m9, INTERP_SHIFT_PS6683
+6684
+ packssdw m8, m96685
+ pshufb m8, m76686
+ movu [r2], m86687
+6688
+ movu m8, [r0 + r1]6689
+ movu m9, [r0 + r1 + 8]6690
+ movu m10, [r0 + r1 + 16]6691
+6692
+ pshufb m11, m8, m56693
+ pshufb m8, m46694
+ pmaddwd m8, m06695
+ pmaddwd m11, m16696
+ paddd m8, m116697
+ pshufb m12, m9, m56698
+ pshufb m9, m46699
+ pmaddwd m11, m12, m36700
+ pmaddwd m14, m9, m26701
+ paddd m11, m146702
+ paddd m8, m116703
+ paddd m8, m66704
+ psrad m8, INTERP_SHIFT_PS6705
+6706
+ pshufb m13, m10, m56707
+ pshufb m10, m46708
+ pmaddwd m9, m06709
+ pmaddwd m12, m16710
+ paddd m9, m126711
+ pmaddwd m12, m13, m36712
+ pmaddwd m14, m10, m26713
+ paddd m12, m146714
+ paddd m9, m126715
+ paddd m9, m66716
+ psrad m9, INTERP_SHIFT_PS6717
+6718
+ packssdw m8, m96719
+ pshufb m8, m76720
+ movu [r2 + r3],m86721
+6722
+ movu m8, [r0 + 2 * r1]6723
+ movu m9, [r0 + 2 * r1 + 8]6724
+ movu m10, [r0 + 2 * r1 + 16]6725
+6726
+ pshufb m11, m8, m56727
+ pshufb m8, m46728
+ pmaddwd m8, m06729
+ pmaddwd m11, m16730
+ paddd m8, m116731
+ pshufb m12, m9, m56732
+ pshufb m9, m46733
+ pmaddwd m11, m12, m36734
+ pmaddwd m14, m9, m26735
+ paddd m11, m146736
+ paddd m8, m116737
+ paddd m8, m66738
+ psrad m8, INTERP_SHIFT_PS6739
+6740
+ pshufb m13, m10, m56741
+ pshufb m10, m46742
+ pmaddwd m9, m06743
+ pmaddwd m12, m16744
+ paddd m9, m126745
+ pmaddwd m13, m36746
+ pmaddwd m10, m26747
+ paddd m10, m136748
+ paddd m9, m106749
+ paddd m9, m66750
+ psrad m9, INTERP_SHIFT_PS6751
+6752
+ packssdw m8, m96753
+ pshufb m8, m76754
+ movu [r2 + 2 * r3], m86755
+6756
+ movu m8, [r0 + r6]6757
+ movu m9, [r0 + r6 + 8]6758
+ movu m10, [r0 + r6 + 16]6759
+6760
+ pshufb m11, m8, m56761
+ pshufb m8, m46762
+ pmaddwd m8, m06763
+ pmaddwd m11, m16764
+ paddd m8, m116765
+ pshufb m12, m9, m56766
+ pshufb m9, m46767
+ pmaddwd m11, m12, m36768
+ pmaddwd m14, m9, m26769
+ paddd m11, m146770
+ paddd m8, m116771
+ paddd m8, m66772
+ psrad m8, INTERP_SHIFT_PS6773
+6774
+ pshufb m13, m10, m56775
+ pshufb m10, m46776
+ pmaddwd m9, m06777
+ pmaddwd m12, m16778
+ paddd m9, m126779
+ pmaddwd m12, m13, m36780
+ pmaddwd m14, m10, m26781
+ paddd m12, m146782
+ paddd m9, m126783
+ paddd m9, m66784
+ psrad m9, INTERP_SHIFT_PS6785
+6786
+ packssdw m8, m96787
+ pshufb m8, m76788
+ movu [r2 + r7],m86789
+6790
+ movu ym8, [r0 + mmsize]6791
+ vinserti32x8 m8, [r0 + r1 + mmsize], 16792
+ movu ym9, [r0 + mmsize + 8]6793
+ vinserti32x8 m9, [r0 + r1 + mmsize + 8], 16794
+ movu ym10, [r0 + mmsize + 16]6795
+ vinserti32x8 m10, [r0 + r1 + mmsize + 16], 16796
+6797
+ pshufb m11, m8, m56798
+ pshufb m8, m46799
+ pmaddwd m8, m06800
+ pmaddwd m11, m16801
+ paddd m8, m116802
+ pshufb m12, m9, m56803
+ pshufb m9, m46804
+ pmaddwd m11, m12, m36805
+ pmaddwd m14, m9, m26806
+ paddd m11, m146807
+ paddd m8, m116808
+ paddd m8, m66809
+ psrad m8, INTERP_SHIFT_PS6810
+6811
+ pshufb m13, m10, m56812
+ pshufb m10, m46813
+ pmaddwd m9, m06814
+ pmaddwd m12, m16815
+ paddd m9, m126816
+ pmaddwd m13, m36817
+ pmaddwd m10, m26818
+ paddd m10, m136819
+ paddd m9, m106820
+ paddd m9, m66821
+ psrad m9, INTERP_SHIFT_PS6822
+6823
+ packssdw m8, m96824
+ pshufb m8, m76825
+ movu [r2 + mmsize], ym86826
+ vextracti32x8 [r2 + r3 + mmsize], m8, 16827
+6828
+ movu ym8, [r0 + 2 * r1 + mmsize]6829
+ vinserti32x8 m8, [r0 + r6 + mmsize], 16830
+ movu ym9, [r0 + 2 * r1 + mmsize + 8]6831
+ vinserti32x8 m9, [r0 + r6 + mmsize + 8], 16832
+ movu ym10, [r0 + 2 * r1 + mmsize + 16]6833
+ vinserti32x8 m10, [r0 + r6 + mmsize + 16], 16834
+6835
+ pshufb m11, m8, m56836
+ pshufb m8, m46837
+ pmaddwd m8, m06838
+ pmaddwd m11, m16839
+ paddd m8, m116840
+ pshufb m12, m9, m56841
+ pshufb m9, m46842
+ pmaddwd m11, m12, m36843
+ pmaddwd m14, m9, m26844
+ paddd m11, m146845
+ paddd m8, m116846
+ paddd m8, m66847
+ psrad m8, INTERP_SHIFT_PS6848
+6849
+ pshufb m13, m10, m56850
+ pshufb m10, m46851
+ pmaddwd m9, m06852
+ pmaddwd m12, m16853
+ paddd m9, m126854
+ pmaddwd m12, m13, m36855
+ pmaddwd m14, m10, m26856
+ paddd m12, m146857
+ paddd m9, m126858
+ paddd m9, m66859
+ psrad m9, INTERP_SHIFT_PS6860
+6861
+ packssdw m8, m96862
+ pshufb m8, m76863
+ movu [r2 + 2 * r3 + mmsize], ym86864
+ vextracti32x8 [r2 + r7 + mmsize], m8, 16865
+%endmacro6866
+6867
+%macro PROCESS_IPFILTER_LUMA_PS_48x3_AVX512 06868
+ movu m8, [r0]6869
+ movu m9, [r0 + 8]6870
+ movu m10, [r0 + 16]6871
+6872
+ pshufb m11, m8, m56873
+ pshufb m8, m46874
+ pmaddwd m8, m06875
+ pmaddwd m11, m16876
+ paddd m8, m116877
+ pshufb m12, m9, m56878
+ pshufb m9, m46879
+ pmaddwd m11, m12, m36880
+ pmaddwd m14, m9, m26881
+ paddd m11, m146882
+ paddd m8, m116883
+ paddd m8, m66884
+ psrad m8, INTERP_SHIFT_PS6885
+6886
+ pshufb m13, m10, m56887
+ pshufb m10, m46888
+ pmaddwd m9, m06889
+ pmaddwd m12, m16890
+ paddd m9, m126891
+ pmaddwd m13, m36892
+ pmaddwd m10, m26893
+ paddd m10, m136894
+ paddd m9, m106895
+ paddd m9, m66896
+ psrad m9, INTERP_SHIFT_PS6897
+6898
+ packssdw m8, m96899
+ pshufb m8, m76900
+ movu [r2], m86901
+6902
+ movu m8, [r0 + r1]6903
+ movu m9, [r0 + r1 + 8]6904
+ movu m10, [r0 + r1 + 16]6905
+6906
+ pshufb m11, m8, m56907
+ pshufb m8, m46908
+ pmaddwd m8, m06909
+ pmaddwd m11, m16910
+ paddd m8, m116911
+ pshufb m12, m9, m56912
+ pshufb m9, m46913
+ pmaddwd m11, m12, m36914
+ pmaddwd m14, m9, m26915
+ paddd m11, m146916
+ paddd m8, m116917
+ paddd m8, m66918
+ psrad m8, INTERP_SHIFT_PS6919
+6920
+ pshufb m13, m10, m56921
+ pshufb m10, m46922
+ pmaddwd m9, m06923
+ pmaddwd m12, m16924
+ paddd m9, m126925
+ pmaddwd m12, m13, m36926
+ pmaddwd m14, m10, m26927
+ paddd m12, m146928
+ paddd m9, m126929
+ paddd m9, m66930
+ psrad m9, INTERP_SHIFT_PS6931
+6932
+ packssdw m8, m96933
+ pshufb m8, m76934
+ movu [r2 + r3],m86935
+6936
+ movu m8, [r0 + 2 * r1]6937
+ movu m9, [r0 + 2 * r1 + 8]6938
+ movu m10, [r0 + 2 * r1 + 16]6939
+6940
+ pshufb m11, m8, m56941
+ pshufb m8, m46942
+ pmaddwd m8, m06943
+ pmaddwd m11, m16944
+ paddd m8, m116945
+ pshufb m12, m9, m56946
+ pshufb m9, m46947
+ pmaddwd m11, m12, m36948
+ pmaddwd m14, m9, m26949
+ paddd m11, m146950
+ paddd m8, m116951
+ paddd m8, m66952
+ psrad m8, INTERP_SHIFT_PS6953
+6954
+ pshufb m13, m10, m56955
+ pshufb m10, m46956
+ pmaddwd m9, m06957
+ pmaddwd m12, m16958
+ paddd m9, m126959
+ pmaddwd m13, m36960
+ pmaddwd m10, m26961
+ paddd m10, m136962
+ paddd m9, m106963
+ paddd m9, m66964
+ psrad m9, INTERP_SHIFT_PS6965
+6966
+ packssdw m8, m96967
+ pshufb m8, m76968
+ movu [r2 + 2 * r3], m86969
+6970
+ movu ym8, [r0 + mmsize]6971
+ vinserti32x8 m8, [r0 + r1 + mmsize], 16972
+ movu ym9, [r0 + mmsize + 8]6973
+ vinserti32x8 m9, [r0 + r1 + mmsize + 8], 16974
+ movu ym10, [r0 + mmsize + 16]6975
+ vinserti32x8 m10, [r0 + r1 + mmsize + 16], 16976
+6977
+ pshufb m11, m8, m56978
+ pshufb m8, m46979
+ pmaddwd m8, m06980
+ pmaddwd m11, m16981
+ paddd m8, m116982
+ pshufb m12, m9, m56983
+ pshufb m9, m46984
+ pmaddwd m11, m12, m36985
+ pmaddwd m14, m9, m26986
+ paddd m11, m146987
+ paddd m8, m116988
+ paddd m8, m66989
+ psrad m8, INTERP_SHIFT_PS6990
+6991
+ pshufb m13, m10, m56992
+ pshufb m10, m46993
+ pmaddwd m9, m06994
+ pmaddwd m12, m16995
+ paddd m9, m126996
+ pmaddwd m13, m36997
+ pmaddwd m10, m26998
+ paddd m10, m136999
+ paddd m9, m107000
+ paddd m9, m67001
+ psrad m9, INTERP_SHIFT_PS7002
+7003
+ packssdw m8, m97004
+ pshufb m8, m77005
+ movu [r2 + mmsize], ym87006
+ vextracti32x8 [r2 + r3 + mmsize], m8, 17007
+7008
+ movu ym8, [r0 + 2 * r1 + mmsize]7009
+ movu ym9, [r0 + 2 * r1 + mmsize + 8]7010
+ movu ym10, [r0 + 2 * r1 + mmsize + 16]7011
+7012
+ pshufb ym11, ym8, ym57013
+ pshufb ym8, ym47014
+ pmaddwd ym8, ym07015
+ pmaddwd ym11, ym17016
+ paddd ym8, ym117017
+ pshufb ym12, ym9, ym57018
+ pshufb ym9, ym47019
+ pmaddwd ym11, ym12, ym37020
+ pmaddwd ym14, ym9, ym27021
+ paddd ym11, ym147022
+ paddd ym8, ym117023
+ paddd ym8, ym67024
+ psrad ym8, INTERP_SHIFT_PS7025
+7026
+ pshufb ym13, ym10, ym57027
+ pshufb ym10, ym47028
+ pmaddwd ym9, ym07029
+ pmaddwd ym12, ym17030
+ paddd ym9, ym127031
+ pmaddwd ym12, ym13, ym37032
+ pmaddwd ym14, ym10, ym27033
+ paddd ym12, ym147034
+ paddd ym9, ym127035
+ paddd ym9, ym67036
+ psrad ym9, INTERP_SHIFT_PS7037
+7038
+ packssdw ym8, ym97039
+ pshufb ym8, ym77040
+ movu [r2 + 2 * r3 + mmsize], ym87041
+%endmacro7042
+7043
+%if ARCH_X86_647044
+INIT_ZMM avx5127045
+cglobal interp_8tap_horiz_ps_48x64, 4,9,157046
+ shl r1d, 17047
+ shl r3d, 17048
+ mov r4d, r4m7049
+ mov r5d, r5m7050
+ shl r4d, 67051
+ lea r6, [3 * r1]7052
+ lea r7, [3 * r3]7053
+%ifdef PIC7054
+ lea r8, [tab_LumaCoeffH_avx512]7055
+ vpbroadcastd m0, [r8 + r4]7056
+ vpbroadcastd m1, [r8 + r4 + 4]7057
+ vpbroadcastd m2, [r8 + r4 + 8]7058
+ vpbroadcastd m3, [r8 + r4 + 12]7059
+%else7060
+ vpbroadcastd m0, [tab_LumaCoeffH_avx512 + r4]7061
+ vpbroadcastd m1, [tab_LumaCoeffH_avx512 + r4 + 4]7062
+ vpbroadcastd m2, [tab_LumaCoeffH_avx512 + r4 + 8]7063
+ vpbroadcastd m3, [tab_LumaCoeffH_avx512 + r4 + 12]7064
+%endif7065
+ vbroadcasti32x8 m4, [interp8_hpp_shuf1_load_avx512]7066
+ vbroadcasti32x8 m5, [interp8_hpp_shuf2_load_avx512]7067
+ vbroadcasti32x4 m6, [INTERP_OFFSET_PS]7068
+ vbroadcasti32x8 m7, [interp8_hpp_shuf1_store_avx512]7069
+7070
+ sub r0, 67071
+ mov r4d, 647072
+ test r5d, r5d7073
+ jz .loop7074
+ lea r6, [r1 * 3]7075
+ sub r0, r67076
+ add r4d, 77077
+ PROCESS_IPFILTER_LUMA_PS_48x4_AVX5127078
+ lea r0, [r0 + r6]7079
+ lea r2, [r2 + r7]7080
+ sub r4d, 37081
+7082
+.loop:7083
+ PROCESS_IPFILTER_LUMA_PS_48x4_AVX5127084
+ lea r0, [r0 + 4 * r1]7085
+ lea r2, [r2 + 4 * r3]7086
+ sub r4d, 47087
+ jnz .loop7088
+ RET7089
+%endif7090
+7091
+%macro PROCESS_IPFILTER_LUMA_PS_24x4_AVX512 07092
+ ; register map7093
+ ; m0 , m1, m2, m3 - interpolate coeff table7094
+ ; m4 , m5 - load shuffle order table7095
+ ; m6 - INTERP_OFFSET_PS7096
+ ; m7 - store shuffle order table7097
+7098
+ PROCESS_IPFILTER_LUMA_PS_16x4_AVX5127099
+7100
+ movu xm8, [r0 + mmsize/2]7101
+ movu xm9, [r0 + mmsize/2 + 8]7102
+ movu xm10, [r0 + mmsize/2 + 16]7103
+7104
+ vinserti32x4 m8, [r0 + r1 + mmsize/2], 17105
+ vinserti32x4 m9, [r0 + r1 + mmsize/2 + 8], 17106
+ vinserti32x4 m10, [r0 + r1 + mmsize/2 + 16], 17107
+7108
+ vinserti32x4 m8, [r0 + 2 * r1 + mmsize/2], 27109
+ vinserti32x4 m9, [r0 + 2 * r1 + mmsize/2 + 8], 27110
+ vinserti32x4 m10, [r0 + 2 * r1 + mmsize/2 + 16], 27111
+7112
+ vinserti32x4 m8, [r0 + r6 + mmsize/2], 37113
+ vinserti32x4 m9, [r0 + r6 + mmsize/2 + 8], 37114
+ vinserti32x4 m10, [r0 + r6 + mmsize/2 + 16], 37115
+7116
+ pshufb m11, m8, m57117
+ pshufb m8, m47118
+ pmaddwd m8, m07119
+ pmaddwd m11, m17120
+ paddd m8, m117121
+ pshufb m12, m9, m57122
+ pshufb m9, m47123
+ pmaddwd m11, m12, m37124
+ pmaddwd m14, m9, m27125
+ paddd m11, m147126
+7127
+ paddd m8, m117128
+ paddd m8, m67129
+ psrad m8, INTERP_SHIFT_PS7130
+7131
+ pshufb m13, m10, m57132
+ pshufb m10, m47133
+ pmaddwd m9, m07134
+ pmaddwd m12, m17135
+ paddd m9, m127136
+ pmaddwd m13, m37137
+ pmaddwd m10, m27138
+ paddd m10, m137139
+7140
+ paddd m9, m107141
+ paddd m9, m67142
+ psrad m9, INTERP_SHIFT_PS7143
+7144
+ packssdw m8, m97145
+ pshufb m8, m77146
+ movu [r2 + mmsize/2], xm87147
+ vextracti32x4 [r2 + r3 + mmsize/2], m8, 17148
+ vextracti32x4 [r2 + 2 * r3 + mmsize/2], m8, 27149
+ vextracti32x4 [r2 + r7 + mmsize/2], m8, 37150
+%endmacro7151
+7152
+%macro PROCESS_IPFILTER_LUMA_PS_24x3_AVX512 07153
+7154
+ PROCESS_IPFILTER_LUMA_PS_16x3_AVX5127155
+7156
+ movu xm8, [r0 + mmsize/2]7157
+ movu xm9, [r0 + mmsize/2 + 8]7158
+ movu xm10, [r0 + mmsize/2 + 16]7159
+7160
+ vinserti32x4 m8, [r0 + r1 + mmsize/2], 17161
+ vinserti32x4 m9, [r0 + r1 + mmsize/2 + 8], 17162
+ vinserti32x4 m10, [r0 + r1 + mmsize/2 + 16], 17163
+7164
+ vinserti32x4 m8, [r0 + 2 * r1 + mmsize/2], 27165
+ vinserti32x4 m9, [r0 + 2 * r1 + mmsize/2 + 8], 27166
+ vinserti32x4 m10, [r0 + 2 * r1 + mmsize/2 + 16], 27167
+7168
+ pshufb m11, m8, m57169
+ pshufb m8, m47170
+ pmaddwd m8, m07171
+ pmaddwd m11, m17172
+ paddd m8, m117173
+ pshufb m12, m9, m57174
+ pshufb m9, m47175
+ pmaddwd m11, m12, m37176
+ pmaddwd m14, m9, m27177
+ paddd m11, m147178
+7179
+ paddd m8, m117180
+ paddd m8, m67181
+ psrad m8, INTERP_SHIFT_PS7182
+7183
+ pshufb m13, m10, m57184
+ pshufb m10, m47185
+ pmaddwd m9, m07186
+ pmaddwd m12, m17187
+ paddd m9, m127188
+ pmaddwd m13, m37189
+ pmaddwd m10, m27190
+ paddd m10, m137191
+7192
+ paddd m9, m107193
+ paddd m9, m67194
+ psrad m9, INTERP_SHIFT_PS7195
+7196
+ packssdw m8, m97197
+ pshufb m8, m77198
+ movu [r2 + mmsize/2], xm87199
+ vextracti32x4 [r2 + r3 + mmsize/2], m8, 17200
+ vextracti32x4 [r2 + 2 * r3 + mmsize/2], m8, 27201
+%endmacro7202
+7203
+%if ARCH_X86_647204
+INIT_ZMM avx5127205
+cglobal interp_8tap_horiz_ps_24x32, 4, 9, 157206
+ shl r1d, 17207
+ shl r3d, 17208
+ mov r4d, r4m7209
+ mov r5d, r5m7210
+ shl r4d, 67211
+7212
+ lea r6, [3 * r1]7213
+ lea r7, [3 * r3]7214
+7215
+%ifdef PIC7216
+ lea r8, [tab_LumaCoeffH_avx512]7217
+ vpbroadcastd m0, [r8 + r4]7218
+ vpbroadcastd m1, [r8 + r4 + 4]7219
+ vpbroadcastd m2, [r8 + r4 + 8]7220
+ vpbroadcastd m3, [r8 + r4 + 12]7221
+%else7222
+ vpbroadcastd m0, [tab_LumaCoeffH_avx512 + r4]7223
+ vpbroadcastd m1, [tab_LumaCoeffH_avx512 + r4 + 4]7224
+ vpbroadcastd m2, [tab_LumaCoeffH_avx512 + r4 + 8]7225
+ vpbroadcastd m3, [tab_LumaCoeffH_avx512 + r4 + 12]7226
+%endif7227
+ vbroadcasti32x8 m4, [interp8_hpp_shuf1_load_avx512]7228
+ vbroadcasti32x8 m5, [interp8_hpp_shuf2_load_avx512]7229
+ vbroadcasti32x4 m6, [INTERP_OFFSET_PS]7230
+ vbroadcasti32x8 m7, [interp8_hpp_shuf1_store_avx512]7231
+7232
+ sub r0, 67233
+ mov r4d, 327234
+ test r5d, r5d7235
+ jz .loop7236
+ sub r0, r67237
+ add r4d, 77238
+ PROCESS_IPFILTER_LUMA_PS_24x3_AVX5127239
+ lea r0, [r0 + r6]7240
+ lea r2, [r2 + r7]7241
+ sub r4d, 37242
+7243
+.loop:7244
+ PROCESS_IPFILTER_LUMA_PS_24x4_AVX5127245
+ lea r0, [r0 + 4 * r1]7246
+ lea r2, [r2 + 4 * r3]7247
+ sub r4d, 47248
+ jnz .loop7249
+ RET7250
+%endif7251
+%macro PROCESS_IPFILTER_LUMA_PS_8x4_AVX512 07252
+ ; register map7253
+ ; m0 , m1, m2, m3 - interpolate coeff table7254
+ ; m4 , m5 - load shuffle order table7255
+ ; m6 - INTERP_OFFSET_PS7256
+ ; m7 - store shuffle order table7257
+7258
+ movu xm8, [r0]7259
+ movu xm9, [r0 + 8]7260
+ movu xm10, [r0 + 16]7261
+7262
+ vinserti32x4 m8, [r0 + r1], 17263
+ vinserti32x4 m9, [r0 + r1 + 8], 17264
+ vinserti32x4 m10, [r0 + r1 + 16], 17265
+7266
+ vinserti32x4 m8, [r0 + 2 * r1], 27267
+ vinserti32x4 m9, [r0 + 2 * r1 + 8], 27268
+ vinserti32x4 m10, [r0 + 2 * r1 + 16], 27269
+7270
+ vinserti32x4 m8, [r0 + r6], 37271
+ vinserti32x4 m9, [r0 + r6 + 8], 37272
+ vinserti32x4 m10, [r0 + r6 + 16], 37273
+7274
+ pshufb m11, m8, m57275
+ pshufb m8, m47276
+ pmaddwd m8, m07277
+ pmaddwd m11, m17278
+ paddd m8, m117279
+ pshufb m12, m9, m57280
+ pshufb m9, m47281
+ pmaddwd m11, m12, m37282
+ pmaddwd m14, m9, m27283
+ paddd m11, m147284
+7285
+ paddd m8, m117286
+ paddd m8, m67287
+ psrad m8, INTERP_SHIFT_PS7288
+7289
+ pshufb m13, m10, m57290
+ pshufb m10, m47291
+ pmaddwd m9, m07292
+ pmaddwd m12, m17293
+ paddd m9, m127294
+ pmaddwd m13, m37295
+ pmaddwd m10, m27296
+ paddd m10, m137297
+7298
+ paddd m9, m107299
+ paddd m9, m67300
+ psrad m9, INTERP_SHIFT_PS7301
+7302
+ packssdw m8, m97303
+ pshufb m8, m77304
+ movu [r2], xm87305
+ vextracti32x4 [r2 + r3], m8, 17306
+ vextracti32x4 [r2 + 2 * r3], m8, 27307
+ vextracti32x4 [r2 + r7], m8, 37308
+%endmacro7309
+7310
+%macro PROCESS_IPFILTER_LUMA_PS_8x3_AVX512 07311
+ movu xm8, [r0]7312
+ movu xm9, [r0 + 8]7313
+ movu xm10, [r0 + 16]7314
+7315
+ vinserti32x4 m8, [r0 + r1], 17316
+ vinserti32x4 m9, [r0 + r1 + 8], 17317
+ vinserti32x4 m10, [r0 + r1 + 16], 17318
+7319
+ vinserti32x4 m8, [r0 + 2 * r1], 27320
+ vinserti32x4 m9, [r0 + 2 * r1 + 8], 27321
+ vinserti32x4 m10, [r0 + 2 * r1 + 16], 27322
+7323
+ pshufb m11, m8, m57324
+ pshufb m8, m47325
+ pmaddwd m8, m07326
+ pmaddwd m11, m17327
+ paddd m8, m117328
+ pshufb m12, m9, m57329
+ pshufb m9, m47330
+ pmaddwd m11, m12, m37331
+ pmaddwd m14, m9, m27332
+ paddd m11, m147333
+7334
+ paddd m8, m117335
+ paddd m8, m67336
+ psrad m8, INTERP_SHIFT_PS7337
+7338
+ pshufb m13, m10, m57339
+ pshufb m10, m47340
+ pmaddwd m9, m07341
+ pmaddwd m12, m17342
+ paddd m9, m127343
+ pmaddwd m13, m37344
+ pmaddwd m10, m27345
+ paddd m10, m137346
+7347
+ paddd m9, m107348
+ paddd m9, m67349
+ psrad m9, INTERP_SHIFT_PS7350
+7351
+ packssdw m8, m97352
+ pshufb m8, m77353
+ movu [r2], xm87354
+ vextracti32x4 [r2 + r3], m8, 17355
+ vextracti32x4 [r2 + 2 * r3], m8, 27356
+%endmacro7357
+7358
+%macro IPFILTER_LUMA_PS_AVX512_8xN 17359
+INIT_ZMM avx5127360
+cglobal interp_8tap_horiz_ps_8x%1, 4, 9, 157361
+ shl r1d, 17362
+ shl r3d, 17363
+ mov r4d, r4m7364
+ mov r5d, r5m7365
+ shl r4d, 67366
+7367
+ lea r6, [3 * r1]7368
+ lea r7, [3 * r3]7369
+7370
+%ifdef PIC7371
+ lea r8, [tab_LumaCoeffH_avx512]7372
+ vpbroadcastd m0, [r8 + r4]7373
+ vpbroadcastd m1, [r8 + r4 + 4]7374
+ vpbroadcastd m2, [r8 + r4 + 8]7375
+ vpbroadcastd m3, [r8 + r4 + 12]7376
+%else7377
+ vpbroadcastd m0, [tab_LumaCoeffH_avx512 + r4]7378
+ vpbroadcastd m1, [tab_LumaCoeffH_avx512 + r4 + 4]7379
+ vpbroadcastd m2, [tab_LumaCoeffH_avx512 + r4 + 8]7380
+ vpbroadcastd m3, [tab_LumaCoeffH_avx512 + r4 + 12]7381
+%endif7382
+ vbroadcasti32x8 m4, [interp8_hpp_shuf1_load_avx512]7383
+ vbroadcasti32x8 m5, [interp8_hpp_shuf2_load_avx512]7384
+ vbroadcasti32x4 m6, [INTERP_OFFSET_PS]7385
+ vbroadcasti32x8 m7, [interp8_hpp_shuf1_store_avx512]7386
+7387
+ sub r0, 67388
+ mov r4d, %17389
+ test r5d, r5d7390
+ jz .loop7391
+ sub r0, r67392
+ add r4d, 77393
+ PROCESS_IPFILTER_LUMA_PS_8x3_AVX5127394
+ lea r0, [r0 + r6]7395
+ lea r2, [r2 + r7]7396
+ sub r4d, 37397
+7398
+.loop:7399
+ PROCESS_IPFILTER_LUMA_PS_8x4_AVX5127400
+ lea r0, [r0 + 4 * r1]7401
+ lea r2, [r2 + 4 * r3]7402
+ sub r4d, 47403
+ jnz .loop7404
+ RET7405
+%endmacro7406
+7407
+%if ARCH_X86_647408
+ IPFILTER_LUMA_PS_AVX512_8xN 47409
+ IPFILTER_LUMA_PS_AVX512_8xN 87410
+ IPFILTER_LUMA_PS_AVX512_8xN 167411
+ IPFILTER_LUMA_PS_AVX512_8xN 327412
+%endif7413
+7414
+;-------------------------------------------------------------------------------------------------------------7415
+;avx512 luma_hps code end7416
+;-------------------------------------------------------------------------------------------------------------7417
+;-------------------------------------------------------------------------------------------------------------7418
+;avx512 luma_vss and luma_vsp code start7419
+;-------------------------------------------------------------------------------------------------------------7420
+%macro PROCESS_LUMA_VERT_S_8x8_AVX512 17421
+ lea r6, [r0 + 4 * r1]7422
+ movu xm1, [r0] ;0 row7423
+ vinserti32x4 m1, [r0 + 2 * r1], 17424
+ vinserti32x4 m1, [r0 + 4 * r1], 27425
+ vinserti32x4 m1, [r6 + 2 * r1], 37426
+ movu xm3, [r0 + r1] ;1 row7427
+ vinserti32x4 m3, [r0 + r7], 17428
+ vinserti32x4 m3, [r6 + r1], 27429
+ vinserti32x4 m3, [r6 + r7], 37430
+ punpcklwd m0, m1, m37431
+ pmaddwd m0, m157432
+ punpckhwd m1, m37433
+ pmaddwd m1, m157434
+7435
+ movu xm4, [r0 + 2 * r1] ;2 row7436
+ vinserti32x4 m4, [r0 + 4 * r1], 17437
+ vinserti32x4 m4, [r6 + 2 * r1], 27438
+ vinserti32x4 m4, [r6 + 4 * r1], 37439
+ punpcklwd m2, m3, m47440
+ pmaddwd m2, m157441
+ punpckhwd m3, m47442
+ pmaddwd m3, m157443
+7444
+ lea r4, [r6 + 4 * r1]7445
+ movu xm5, [r0 + r7] ;3 row7446
+ vinserti32x4 m5, [r6 + r1], 17447
+ vinserti32x4 m5, [r6 + r7], 27448
+ vinserti32x4 m5, [r4 + r1], 37449
+ punpcklwd m6, m4, m57450
+ pmaddwd m6, m167451
+ punpckhwd m4, m57452
+ pmaddwd m4, m167453
+7454
+ paddd m0, m67455
+ paddd m1, m47456
+7457
+ movu xm4, [r0 + 4 * r1] ;4 row7458
+ vinserti32x4 m4, [r6 + 2 * r1], 17459
+ vinserti32x4 m4, [r6 + 4 * r1], 27460
+ vinserti32x4 m4, [r4 + 2 * r1], 37461
+ punpcklwd m6, m5, m47462
+ pmaddwd m6, m167463
+ punpckhwd m5, m47464
+ pmaddwd m5, m167465
+7466
+ paddd m2, m67467
+ paddd m3, m57468
+7469
+ movu xm11, [r6 + r1] ;5 row7470
+ vinserti32x4 m11, [r6 + r7], 17471
+ vinserti32x4 m11, [r4 + r1], 27472
+ vinserti32x4 m11, [r4 + r7], 37473
+ punpcklwd m8, m4, m117474
+ pmaddwd m8, m177475
+ punpckhwd m4, m117476
+ pmaddwd m4, m177477
+7478
+ movu xm12, [r6 + 2 * r1] ;6 row7479
+ vinserti32x4 m12, [r6 + 4 * r1], 17480
+ vinserti32x4 m12, [r4 + 2 * r1], 27481
+ vinserti32x4 m12, [r4 + 4 * r1], 37482
+ punpcklwd m10, m11, m127483
+ pmaddwd m10, m177484
+ punpckhwd m11, m127485
+ pmaddwd m11, m177486
+7487
+ lea r8, [r4 + 4 * r1]7488
+ movu xm13, [r6 + r7] ;7 row7489
+ vinserti32x4 m13, [r4 + r1], 17490
+ vinserti32x4 m13, [r4 + r7], 27491
+ vinserti32x4 m13, [r8 + r1], 37492
+ punpcklwd m14, m12, m137493
+ pmaddwd m14, m187494
+ punpckhwd m12, m137495
+ pmaddwd m12, m187496
+7497
+ paddd m8, m147498
+ paddd m4, m127499
+ paddd m0, m87500
+ paddd m1, m47501
+7502
+ movu xm12, [r6 + 4 * r1] ; 8 row7503
+ vinserti32x4 m12, [r4 + 2 * r1], 17504
+ vinserti32x4 m12, [r4 + 4 * r1], 27505
+ vinserti32x4 m12, [r8 + 2 * r1], 37506
+ punpcklwd m14, m13, m127507
+ pmaddwd m14, m187508
+ punpckhwd m13, m127509
+ pmaddwd m13, m187510
+7511
+ paddd m10, m147512
+ paddd m11, m137513
+ paddd m2, m107514
+ paddd m3, m117515
+7516
+%ifidn %1, sp7517
+ paddd m0, m197518
+ paddd m1, m197519
+ paddd m2, m197520
+ paddd m3, m197521
+7522
+ psrad m0, INTERP_SHIFT_SP7523
+ psrad m1, INTERP_SHIFT_SP7524
+ psrad m2, INTERP_SHIFT_SP7525
+ psrad m3, INTERP_SHIFT_SP7526
+7527
+ packssdw m0, m17528
+ packssdw m2, m37529
+ CLIPW2 m0, m2, m20, m217530
+%else7531
+ psrad m0, 67532
+ psrad m1, 67533
+ psrad m2, 67534
+ psrad m3, 67535
+7536
+ packssdw m0, m17537
+ packssdw m2, m37538
+%endif7539
+7540
+ movu [r2], xm07541
+ movu [r2 + r3], xm27542
+ vextracti32x4 [r2 + 2 * r3], m0, 17543
+ vextracti32x4 [r2 + r5], m2, 17544
+ lea r2, [r2 + 4 * r3]7545
+ vextracti32x4 [r2], m0, 27546
+ vextracti32x4 [r2 + r3], m2, 27547
+ vextracti32x4 [r2 + 2 * r3], m0, 37548
+ vextracti32x4 [r2 + r5], m2, 37549
+%endmacro7550
+;-----------------------------------------------------------------------------------------------------------------7551
+; void interp_8tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)7552
+;-----------------------------------------------------------------------------------------------------------------7553
+%macro FILTER_VER_S_LUMA_8xN_AVX512 27554
+INIT_ZMM avx5127555
+cglobal interp_8tap_vert_%1_8x%2, 5, 9, 227556
+ add r1d, r1d7557
+ add r3d, r3d7558
+ lea r7, [3 * r1]7559
+ sub r0, r77560
+ shl r4d, 87561
+%ifdef PIC7562
+ lea r5, [tab_LumaCoeffVer_avx512]7563
+ mova m15, [r5 + r4]7564
+ mova m16, [r5 + r4 + 1 * mmsize]7565
+ mova m17, [r5 + r4 + 2 * mmsize]7566
+ mova m18, [r5 + r4 + 3 * mmsize]7567
+%else7568
+ lea r5, [tab_LumaCoeffVer_avx512 + r4]7569
+ mova m15, [r5]7570
+ mova m16, [r5 + 1 * mmsize]7571
+ mova m17, [r5 + 2 * mmsize]7572
+ mova m18, [r5 + 3 * mmsize]7573
+%endif7574
+%ifidn %1, sp7575
+ vbroadcasti32x4 m19, [INTERP_OFFSET_SP]7576
+ pxor m20, m207577
+ vbroadcasti32x8 m21, [pw_pixel_max]7578
+%endif7579
+ lea r5, [3 * r3]7580
+7581
+%rep %2/8 - 17582
+ PROCESS_LUMA_VERT_S_8x8_AVX512 %17583
+ lea r0, [r4]7584
+ lea r2, [r2 + 4 * r3]7585
+%endrep7586
+ PROCESS_LUMA_VERT_S_8x8_AVX512 %17587
+ RET7588
+%endmacro7589
+7590
+%if ARCH_X86_647591
+ FILTER_VER_S_LUMA_8xN_AVX512 ss, 87592
+ FILTER_VER_S_LUMA_8xN_AVX512 ss, 167593
+ FILTER_VER_S_LUMA_8xN_AVX512 ss, 327594
+ FILTER_VER_S_LUMA_8xN_AVX512 sp, 87595
+ FILTER_VER_S_LUMA_8xN_AVX512 sp, 167596
+ FILTER_VER_S_LUMA_8xN_AVX512 sp, 327597
+%endif7598
+7599
+%macro PROCESS_LUMA_VERT_S_16x4_AVX512 17600
+ movu ym1, [r0]7601
+ movu ym3, [r0 + r1]7602
+ vinserti32x8 m1, [r0 + 2 * r1], 17603
+ vinserti32x8 m3, [r0 + r7], 17604
+ punpcklwd m0, m1, m37605
+ pmaddwd m0, m157606
+ punpckhwd m1, m37607
+ pmaddwd m1, m157608
+7609
+ lea r6, [r0 + 4 * r1]7610
+ movu ym4, [r0 + 2 * r1]7611
+ vinserti32x8 m4, [r6], 17612
+ punpcklwd m2, m3, m47613
+ pmaddwd m2, m157614
+ punpckhwd m3, m47615
+ pmaddwd m3, m157616
+7617
+ movu ym5, [r0 + r7]7618
+ vinserti32x8 m5, [r6 + r1], 17619
+ punpcklwd m6, m4, m57620
+ pmaddwd m6, m167621
+ punpckhwd m4, m57622
+ pmaddwd m4, m167623
+7624
+ paddd m0, m67625
+ paddd m1, m47626
+7627
+ movu ym4, [r6]7628
+ vinserti32x8 m4, [r6 + 2 * r1], 17629
+ punpcklwd m6, m5, m47630
+ pmaddwd m6, m167631
+ punpckhwd m5, m47632
+ pmaddwd m5, m167633
+7634
+ paddd m2, m67635
+ paddd m3, m57636
+7637
+ movu ym11, [r6 + r1]7638
+ vinserti32x8 m11, [r6 + r7], 17639
+ punpcklwd m8, m4, m117640
+ pmaddwd m8, m177641
+ punpckhwd m4, m117642
+ pmaddwd m4, m177643
+7644
+ movu ym12, [r6 + 2 * r1]7645
+ vinserti32x8 m12, [r6 + 4 * r1], 17646
+ punpcklwd m10, m11, m127647
+ pmaddwd m10, m177648
+ punpckhwd m11, m127649
+ pmaddwd m11, m177650
+7651
+ lea r4, [r6 + 4 * r1]7652
+ movu ym13, [r6 + r7]7653
+ vinserti32x8 m13, [r4 + r1], 17654
+ punpcklwd m14, m12, m137655
+ pmaddwd m14, m187656
+ punpckhwd m12, m137657
+ pmaddwd m12, m187658
+7659
+ paddd m8, m147660
+ paddd m4, m127661
+ paddd m0, m87662
+ paddd m1, m47663
+7664
+ movu ym12, [r6 + 4 * r1]7665
+ vinserti32x8 m12, [r4 + 2 * r1], 17666
+ punpcklwd m14, m13, m127667
+ pmaddwd m14, m187668
+ punpckhwd m13, m127669
+ pmaddwd m13, m187670
+7671
+ paddd m10, m147672
+ paddd m11, m137673
+ paddd m2, m107674
+ paddd m3, m117675
+7676
+%ifidn %1, sp7677
+ paddd m0, m197678
+ paddd m1, m197679
+ paddd m2, m197680
+ paddd m3, m197681
+7682
+ psrad m0, INTERP_SHIFT_SP7683
+ psrad m1, INTERP_SHIFT_SP7684
+ psrad m2, INTERP_SHIFT_SP7685
+ psrad m3, INTERP_SHIFT_SP7686
+7687
+ packssdw m0, m17688
+ packssdw m2, m37689
+ CLIPW2 m0, m2, m20, m217690
+%else7691
+ psrad m0, 67692
+ psrad m1, 67693
+ psrad m2, 67694
+ psrad m3, 67695
+7696
+ packssdw m0, m17697
+ packssdw m2, m37698
+%endif7699
+7700
+ movu [r2], ym07701
+ movu [r2 + r3], ym27702
+ vextracti32x8 [r2 + 2 * r3], m0, 17703
+ vextracti32x8 [r2 + r5], m2, 17704
+%endmacro7705
+;-----------------------------------------------------------------------------------------------------------------7706
+; void interp_8tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)7707
+;-----------------------------------------------------------------------------------------------------------------7708
+%macro FILTER_VER_S_LUMA_16xN_AVX512 27709
+INIT_ZMM avx5127710
+cglobal interp_8tap_vert_%1_16x%2, 5, 8, 227711
+ add r1d, r1d7712
+ add r3d, r3d7713
+ lea r7, [3 * r1]7714
+ sub r0, r77715
+ shl r4d, 87716
+%ifdef PIC7717
+ lea r5, [tab_LumaCoeffVer_avx512]7718
+ mova m15, [r5 + r4]7719
+ mova m16, [r5 + r4 + 1 * mmsize]7720
+ mova m17, [r5 + r4 + 2 * mmsize]7721
+ mova m18, [r5 + r4 + 3 * mmsize]7722
+%else7723
+ lea r5, [tab_LumaCoeffVer_avx512 + r4]7724
+ mova m15, [r5]7725
+ mova m16, [r5 + 1 * mmsize]7726
+ mova m17, [r5 + 2 * mmsize]7727
+ mova m18, [r5 + 3 * mmsize]7728
+%endif7729
+%ifidn %1, sp7730
+ vbroadcasti32x4 m19, [INTERP_OFFSET_SP]7731
+ pxor m20, m207732
+ vbroadcasti32x8 m21, [pw_pixel_max]7733
+%endif7734
+ lea r5, [3 * r3]7735
+%rep %2/4 - 17736
+ PROCESS_LUMA_VERT_S_16x4_AVX512 %17737
+ lea r0, [r0 + 4 * r1]7738
+ lea r2, [r2 + 4 * r3]7739
+%endrep7740
+ PROCESS_LUMA_VERT_S_16x4_AVX512 %17741
+ RET7742
+%endmacro7743
+7744
+%if ARCH_X86_647745
+ FILTER_VER_S_LUMA_16xN_AVX512 ss, 47746
+ FILTER_VER_S_LUMA_16xN_AVX512 ss, 87747
+ FILTER_VER_S_LUMA_16xN_AVX512 ss, 127748
+ FILTER_VER_S_LUMA_16xN_AVX512 ss, 167749
+ FILTER_VER_S_LUMA_16xN_AVX512 ss, 327750
+ FILTER_VER_S_LUMA_16xN_AVX512 ss, 647751
+ FILTER_VER_S_LUMA_16xN_AVX512 sp, 47752
+ FILTER_VER_S_LUMA_16xN_AVX512 sp, 87753
+ FILTER_VER_S_LUMA_16xN_AVX512 sp, 127754
+ FILTER_VER_S_LUMA_16xN_AVX512 sp, 167755
+ FILTER_VER_S_LUMA_16xN_AVX512 sp, 327756
+ FILTER_VER_S_LUMA_16xN_AVX512 sp, 647757
+%endif7758
+7759
+%macro PROCESS_LUMA_VERT_S_24x8_AVX512 17760
+ PROCESS_LUMA_VERT_S_16x4_AVX512 %17761
+ lea r4, [r6 + 4 * r1]7762
+ lea r8, [r4 + 4 * r1]7763
+ movu ym1, [r6]7764
+ movu ym3, [r6 + r1]7765
+ vinserti32x8 m1, [r6 + 2 * r1], 17766
+ vinserti32x8 m3, [r6 + r7], 17767
+ punpcklwd m0, m1, m37768
+ pmaddwd m0, m157769
+ punpckhwd m1, m37770
+ pmaddwd m1, m157771
+7772
+ movu ym4, [r6 + 2 * r1]7773
+ vinserti32x8 m4, [r4], 17774
+ punpcklwd m2, m3, m47775
+ pmaddwd m2, m157776
+ punpckhwd m3, m47777
+ pmaddwd m3, m157778
+7779
+ movu ym5, [r6 + r7]7780
+ vinserti32x8 m5, [r4 + r1], 17781
+ punpcklwd m6, m4, m57782
+ pmaddwd m6, m167783
+ punpckhwd m4, m57784
+ pmaddwd m4, m167785
+7786
+ paddd m0, m67787
+ paddd m1, m47788
+7789
+ movu ym4, [r4]7790
+ vinserti32x8 m4, [r4 + 2 * r1], 17791
+ punpcklwd m6, m5, m47792
+ pmaddwd m6, m167793
+ punpckhwd m5, m47794
+ pmaddwd m5, m167795
+7796
+ paddd m2, m67797
+ paddd m3, m57798
+7799
+ movu ym11, [r4 + r1]7800
+ vinserti32x8 m11, [r4 + r7], 17801
+ punpcklwd m8, m4, m117802
+ pmaddwd m8, m177803
+ punpckhwd m4, m117804
+ pmaddwd m4, m177805
+7806
+ movu ym12, [r4 + 2 * r1]7807
+ vinserti32x8 m12, [r4 + 4 * r1], 17808
+ punpcklwd m10, m11, m127809
+ pmaddwd m10, m177810
+ punpckhwd m11, m127811
+ pmaddwd m11, m177812
+7813
+ movu ym13, [r4 + r7]7814
+ vinserti32x8 m13, [r8 + r1], 17815
+ punpcklwd m14, m12, m137816
+ pmaddwd m14, m187817
+ punpckhwd m12, m137818
+ pmaddwd m12, m187819
+7820
+ paddd m8, m147821
+ paddd m4, m127822
+ paddd m0, m87823
+ paddd m1, m47824
+7825
+ movu ym12, [r4 + 4 * r1]7826
+ vinserti32x8 m12, [r8 + 2 * r1], 17827
+ punpcklwd m14, m13, m127828
+ pmaddwd m14, m187829
+ punpckhwd m13, m127830
+ pmaddwd m13, m187831
+7832
+ paddd m10, m147833
+ paddd m11, m137834
+ paddd m2, m107835
+ paddd m3, m117836
+7837
+%ifidn %1, sp7838
+ paddd m0, m197839
+ paddd m1, m197840
+ paddd m2, m197841
+ paddd m3, m197842
+7843
+ psrad m0, INTERP_SHIFT_SP7844
+ psrad m1, INTERP_SHIFT_SP7845
+ psrad m2, INTERP_SHIFT_SP7846
+ psrad m3, INTERP_SHIFT_SP7847
+7848
+ packssdw m0, m17849
+ packssdw m2, m37850
+ CLIPW2 m0, m2, m20, m217851
+%else7852
+ psrad m0, 67853
+ psrad m1, 67854
+ psrad m2, 67855
+ psrad m3, 67856
+7857
+ packssdw m0, m17858
+ packssdw m2, m37859
+%endif7860
+ lea r9, [r2 + 4 * r3]7861
+ movu [r9], ym07862
+ movu [r9 + r3], ym27863
+ vextracti32x8 [r9 + 2 * r3], m0, 17864
+ vextracti32x8 [r9 + r5], m2, 17865
+7866
+ movu xm1, [r0 + mmsize/2]7867
+ vinserti32x4 m1, [r0 + 2 * r1 + mmsize/2], 17868
+ vinserti32x4 m1, [r0 + 4 * r1 + mmsize/2], 27869
+ vinserti32x4 m1, [r6 + 2 * r1 + mmsize/2], 37870
+ movu xm3, [r0 + r1 + mmsize/2]7871
+ vinserti32x4 m3, [r0 + r7 + mmsize/2], 17872
+ vinserti32x4 m3, [r6 + r1 + mmsize/2], 27873
+ vinserti32x4 m3, [r6 + r7 + mmsize/2], 37874
+ punpcklwd m0, m1, m37875
+ pmaddwd m0, m157876
+ punpckhwd m1, m37877
+ pmaddwd m1, m157878
+7879
+ movu xm4, [r0 + 2 * r1 + mmsize/2]7880
+ vinserti32x4 m4, [r0 + 4 * r1 + mmsize/2], 17881
+ vinserti32x4 m4, [r6 + 2 * r1 + mmsize/2], 27882
+ vinserti32x4 m4, [r6 + 4 * r1 + mmsize/2], 37883
+ punpcklwd m2, m3, m47884
+ pmaddwd m2, m157885
+ punpckhwd m3, m47886
+ pmaddwd m3, m157887
+7888
+ movu xm5, [r0 + r7 + mmsize/2]7889
+ vinserti32x4 m5, [r6 + r1 + mmsize/2], 17890
+ vinserti32x4 m5, [r6 + r7 + mmsize/2], 27891
+ vinserti32x4 m5, [r4 + r1 + mmsize/2], 37892
+ punpcklwd m6, m4, m57893
+ pmaddwd m6, m167894
+ punpckhwd m4, m57895
+ pmaddwd m4, m167896
+7897
+ paddd m0, m67898
+ paddd m1, m47899
+7900
+ movu xm4, [r0 + 4 * r1 + mmsize/2]7901
+ vinserti32x4 m4, [r6 + 2 * r1 + mmsize/2], 17902
+ vinserti32x4 m4, [r6 + 4 * r1 + mmsize/2], 27903
+ vinserti32x4 m4, [r4 + 2 * r1 + mmsize/2], 37904
+ punpcklwd m6, m5, m47905
+ pmaddwd m6, m167906
+ punpckhwd m5, m47907
+ pmaddwd m5, m167908
+7909
+ paddd m2, m67910
+ paddd m3, m57911
+7912
+ movu xm11, [r6 + r1 + mmsize/2]7913
+ vinserti32x4 m11, [r6 + r7 + mmsize/2], 17914
+ vinserti32x4 m11, [r4 + r1 + mmsize/2], 27915
+ vinserti32x4 m11, [r4 + r7 + mmsize/2], 37916
+ punpcklwd m8, m4, m117917
+ pmaddwd m8, m177918
+ punpckhwd m4, m117919
+ pmaddwd m4, m177920
+7921
+ movu xm12, [r6 + 2 * r1 + mmsize/2]7922
+ vinserti32x4 m12, [r6 + 4 * r1 + mmsize/2], 17923
+ vinserti32x4 m12, [r4 + 2 * r1 + mmsize/2], 27924
+ vinserti32x4 m12, [r4 + 4 * r1 + mmsize/2], 37925
+ punpcklwd m10, m11, m127926
+ pmaddwd m10, m177927
+ punpckhwd m11, m127928
+ pmaddwd m11, m177929
+7930
+ movu xm13, [r6 + r7 + mmsize/2]7931
+ vinserti32x4 m13, [r4 + r1 + mmsize/2], 17932
+ vinserti32x4 m13, [r4 + r7 + mmsize/2], 27933
+ vinserti32x4 m13, [r8 + r1 + mmsize/2], 37934
+ punpcklwd m14, m12, m137935
+ pmaddwd m14, m187936
+ punpckhwd m12, m137937
+ pmaddwd m12, m187938
+7939
+ paddd m8, m147940
+ paddd m4, m127941
+ paddd m0, m87942
+ paddd m1, m47943
+7944
+ movu xm12, [r6 + 4 * r1 + mmsize/2]7945
+ vinserti32x4 m12, [r4 + 2 * r1 + mmsize/2], 17946
+ vinserti32x4 m12, [r4 + 4 * r1 + mmsize/2], 27947
+ vinserti32x4 m12, [r8 + 2 * r1 + mmsize/2], 37948
+ punpcklwd m14, m13, m127949
+ pmaddwd m14, m187950
+ punpckhwd m13, m127951
+ pmaddwd m13, m187952
+7953
+ paddd m10, m147954
+ paddd m11, m137955
+ paddd m2, m107956
+ paddd m3, m117957
+7958
+%ifidn %1, sp7959
+ paddd m0, m197960
+ paddd m1, m197961
+ paddd m2, m197962
+ paddd m3, m197963
+7964
+ psrad m0, INTERP_SHIFT_SP7965
+ psrad m1, INTERP_SHIFT_SP7966
+ psrad m2, INTERP_SHIFT_SP7967
+ psrad m3, INTERP_SHIFT_SP7968
+7969
+ packssdw m0, m17970
+ packssdw m2, m37971
+ CLIPW2 m0, m2, m20, m217972
+%else7973
+ psrad m0, 67974
+ psrad m1, 67975
+ psrad m2, 67976
+ psrad m3, 67977
+7978
+ packssdw m0, m17979
+ packssdw m2, m37980
+%endif7981
+7982
+ movu [r2 + mmsize/2], xm07983
+ movu [r2 + r3 + mmsize/2], xm27984
+ vextracti32x4 [r2 + 2 * r3 + mmsize/2], m0, 17985
+ vextracti32x4 [r2 + r5 + mmsize/2], m2, 17986
+ lea r2, [r2 + 4 * r3]7987
+ vextracti32x4 [r2 + mmsize/2], m0, 27988
+ vextracti32x4 [r2 + r3 + mmsize/2], m2, 27989
+ vextracti32x4 [r2 + 2 * r3 + mmsize/2], m0, 37990
+ vextracti32x4 [r2 + r5 + mmsize/2], m2, 37991
+%endmacro7992
+;-----------------------------------------------------------------------------------------------------------------7993
+; void interp_8tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)7994
+;-----------------------------------------------------------------------------------------------------------------7995
+%macro FILTER_VER_S_LUMA_24x32_AVX512 17996
+INIT_ZMM avx5127997
+cglobal interp_8tap_vert_%1_24x32, 5, 10, 227998
+ add r1d, r1d7999
+ add r3d, r3d8000
+ lea r7, [3 * r1]8001
+ sub r0, r78002
+ shl r4d, 88003
+%ifdef PIC8004
+ lea r5, [tab_LumaCoeffVer_avx512]8005
+ mova m15, [r5 + r4]8006
+ mova m16, [r5 + r4 + 1 * mmsize]8007
+ mova m17, [r5 + r4 + 2 * mmsize]8008
+ mova m18, [r5 + r4 + 3 * mmsize]8009
+%else8010
+ lea r5, [tab_LumaCoeffVer_avx512 + r4]8011
+ mova m15, [r5]8012
+ mova m16, [r5 + 1 * mmsize]8013
+ mova m17, [r5 + 2 * mmsize]8014
+ mova m18, [r5 + 3 * mmsize]8015
+%endif8016
+%ifidn %1, sp8017
+ vbroadcasti32x4 m19, [INTERP_OFFSET_SP]8018
+ pxor m20, m208019
+ vbroadcasti32x8 m21, [pw_pixel_max]8020
+%endif8021
+ lea r5, [3 * r3]8022
+8023
+%rep 38024
+ PROCESS_LUMA_VERT_S_24x8_AVX512 %18025
+ lea r0, [r4]8026
+ lea r2, [r2 + 4 * r3]8027
+%endrep8028
+ PROCESS_LUMA_VERT_S_24x8_AVX512 %18029
+ RET8030
+%endmacro8031
+8032
+%if ARCH_X86_648033
+ FILTER_VER_S_LUMA_24x32_AVX512 ss8034
+ FILTER_VER_S_LUMA_24x32_AVX512 sp8035
+%endif8036
+8037
+%macro PROCESS_LUMA_VERT_S_32x2_AVX512 18038
+ movu m1, [r0] ;0 row8039
+ movu m3, [r0 + r1] ;1 row8040
+ punpcklwd m0, m1, m38041
+ pmaddwd m0, m158042
+ punpckhwd m1, m38043
+ pmaddwd m1, m158044
+8045
+ movu m4, [r0 + 2 * r1] ;2 row8046
+ punpcklwd m2, m3, m48047
+ pmaddwd m2, m158048
+ punpckhwd m3, m48049
+ pmaddwd m3, m158050
+8051
+ movu m5, [r0 + r7] ;3 row8052
+ punpcklwd m6, m4, m58053
+ pmaddwd m6, m168054
+ punpckhwd m4, m58055
+ pmaddwd m4, m168056
+8057
+ paddd m0, m68058
+ paddd m1, m48059
+8060
+ movu m4, [r0 + 4 * r1] ;4 row8061
+ punpcklwd m6, m5, m48062
+ pmaddwd m6, m168063
+ punpckhwd m5, m48064
+ pmaddwd m5, m168065
+8066
+ paddd m2, m68067
+ paddd m3, m58068
+8069
+ lea r6, [r0 + 4 * r1]8070
+8071
+ movu m11, [r6 + r1] ;5 row8072
+ punpcklwd m8, m4, m118073
+ pmaddwd m8, m178074
+ punpckhwd m4, m118075
+ pmaddwd m4, m178076
+8077
+ movu m12, [r6 + 2 * r1] ;6 row8078
+ punpcklwd m10, m11, m128079
+ pmaddwd m10, m178080
+ punpckhwd m11, m128081
+ pmaddwd m11, m178082
+8083
+ movu m13, [r6 + r7] ;7 row8084
+ punpcklwd m14, m12, m138085
+ pmaddwd m14, m188086
+ punpckhwd m12, m138087
+ pmaddwd m12, m188088
+8089
+ paddd m8, m148090
+ paddd m4, m128091
+ paddd m0, m88092
+ paddd m1, m48093
+8094
+ movu m12, [r6 + 4 * r1] ; 8 row8095
+ punpcklwd m14, m13, m128096
+ pmaddwd m14, m188097
+ punpckhwd m13, m128098
+ pmaddwd m13, m188099
+8100
+ paddd m10, m148101
+ paddd m11, m138102
+ paddd m2, m108103
+ paddd m3, m118104
+8105
+%ifidn %1, sp8106
+ paddd m0, m198107
+ paddd m1, m198108
+ paddd m2, m198109
+ paddd m3, m198110
+8111
+ psrad m0, INTERP_SHIFT_SP8112
+ psrad m1, INTERP_SHIFT_SP8113
+ psrad m2, INTERP_SHIFT_SP8114
+ psrad m3, INTERP_SHIFT_SP8115
+8116
+ packssdw m0, m18117
+ packssdw m2, m38118
+ CLIPW2 m0, m2, m20, m218119
+%else8120
+ psrad m0, 68121
+ psrad m1, 68122
+ psrad m2, 68123
+ psrad m3, 68124
+8125
+ packssdw m0, m18126
+ packssdw m2, m38127
+%endif8128
+8129
+ movu [r2], m08130
+ movu [r2 + r3], m28131
+%endmacro8132
+;-----------------------------------------------------------------------------------------------------------------8133
+; void interp_4tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)8134
+;-----------------------------------------------------------------------------------------------------------------8135
+%macro FILTER_VER_S_LUMA_32xN_AVX512 28136
+INIT_ZMM avx5128137
+cglobal interp_8tap_vert_%1_32x%2, 5, 8, 228138
+ add r1d, r1d8139
+ add r3d, r3d8140
+ lea r7, [3 * r1]8141
+ sub r0, r78142
+ shl r4d, 88143
+%ifdef PIC8144
+ lea r5, [tab_LumaCoeffVer_avx512]8145
+ mova m15, [r5 + r4]8146
+ mova m16, [r5 + r4 + 1 * mmsize]8147
+ mova m17, [r5 + r4 + 2 * mmsize]8148
+ mova m18, [r5 + r4 + 3 * mmsize]8149
+%else8150
+ lea r5, [tab_LumaCoeffVer_avx512 + r4]8151
+ mova m15, [r5]8152
+ mova m16, [r5 + 1 * mmsize]8153
+ mova m17, [r5 + 2 * mmsize]8154
+ mova m18, [r5 + 3 * mmsize]8155
+%endif8156
+%ifidn %1, sp8157
+ vbroadcasti32x4 m19, [INTERP_OFFSET_SP]8158
+ pxor m20, m208159
+ vbroadcasti32x8 m21, [pw_pixel_max]8160
+%endif8161
+8162
+%rep %2/2 - 18163
+ PROCESS_LUMA_VERT_S_32x2_AVX512 %18164
+ lea r0, [r0 + 2 * r1]8165
+ lea r2, [r2 + 2 * r3]8166
+%endrep8167
+ PROCESS_LUMA_VERT_S_32x2_AVX512 %18168
+ RET8169
+%endmacro8170
+8171
+%if ARCH_X86_648172
+ FILTER_VER_S_LUMA_32xN_AVX512 ss, 88173
+ FILTER_VER_S_LUMA_32xN_AVX512 ss, 168174
+ FILTER_VER_S_LUMA_32xN_AVX512 ss, 328175
+ FILTER_VER_S_LUMA_32xN_AVX512 ss, 248176
+ FILTER_VER_S_LUMA_32xN_AVX512 ss, 648177
+ FILTER_VER_S_LUMA_32xN_AVX512 sp, 88178
+ FILTER_VER_S_LUMA_32xN_AVX512 sp, 168179
+ FILTER_VER_S_LUMA_32xN_AVX512 sp, 328180
+ FILTER_VER_S_LUMA_32xN_AVX512 sp, 248181
+ FILTER_VER_S_LUMA_32xN_AVX512 sp, 648182
+%endif8183
+8184
+%macro PROCESS_LUMA_VERT_S_48x4_AVX512 18185
+ PROCESS_LUMA_VERT_S_32x2_AVX512 %18186
+ movu m1, [r0 + 2 * r1]8187
+ movu m3, [r0 + r7]8188
+ punpcklwd m0, m1, m38189
+ pmaddwd m0, m158190
+ punpckhwd m1, m38191
+ pmaddwd m1, m158192
+8193
+ movu m4, [r0 + 4 * r1]8194
+ punpcklwd m2, m3, m48195
+ pmaddwd m2, m158196
+ punpckhwd m3, m48197
+ pmaddwd m3, m158198
+8199
+ movu m5, [r6 + r1]8200
+ punpcklwd m6, m4, m58201
+ pmaddwd m6, m168202
+ punpckhwd m4, m58203
+ pmaddwd m4, m168204
+8205
+ paddd m0, m68206
+ paddd m1, m48207
+8208
+ lea r4, [r6 + 4 * r1]8209
+8210
+ movu m4, [r6 + 2 * r1]8211
+ punpcklwd m6, m5, m48212
+ pmaddwd m6, m168213
+ punpckhwd m5, m48214
+ pmaddwd m5, m168215
+8216
+ paddd m2, m68217
+ paddd m3, m58218
+8219
+ movu m11, [r6 + r7]8220
+ punpcklwd m8, m4, m118221
+ pmaddwd m8, m178222
+ punpckhwd m4, m118223
+ pmaddwd m4, m178224
+8225
+ movu m12, [r4]8226
+ punpcklwd m10, m11, m128227
+ pmaddwd m10, m178228
+ punpckhwd m11, m128229
+ pmaddwd m11, m178230
+8231
+ movu m13, [r4 + r1]8232
+ punpcklwd m14, m12, m138233
+ pmaddwd m14, m188234
+ punpckhwd m12, m138235
+ pmaddwd m12, m188236
+8237
+ paddd m8, m148238
+ paddd m4, m128239
+ paddd m0, m88240
+ paddd m1, m48241
+8242
+ movu m12, [r4 + 2 * r1]8243
+ punpcklwd m14, m13, m128244
+ pmaddwd m14, m188245
+ punpckhwd m13, m128246
+ pmaddwd m13, m188247
+8248
+ paddd m10, m148249
+ paddd m11, m138250
+ paddd m2, m108251
+ paddd m3, m118252
+8253
+%ifidn %1, sp8254
+ paddd m0, m198255
+ paddd m1, m198256
+ paddd m2, m198257
+ paddd m3, m198258
+8259
+ psrad m0, INTERP_SHIFT_SP8260
+ psrad m1, INTERP_SHIFT_SP8261
+ psrad m2, INTERP_SHIFT_SP8262
+ psrad m3, INTERP_SHIFT_SP8263
+8264
+ packssdw m0, m18265
+ packssdw m2, m38266
+ CLIPW2 m0, m2, m20, m218267
+%else8268
+ psrad m0, 68269
+ psrad m1, 68270
+ psrad m2, 68271
+ psrad m3, 68272
+8273
+ packssdw m0, m18274
+ packssdw m2, m38275
+%endif8276
+8277
+ movu [r2 + 2 * r3], m08278
+ movu [r2 + r5], m28279
+8280
+ movu ym1, [r0 + mmsize]8281
+ movu ym3, [r0 + r1 + mmsize]8282
+ vinserti32x8 m1, [r0 + 2 * r1 + mmsize], 18283
+ vinserti32x8 m3, [r0 + r7 + mmsize], 18284
+ punpcklwd m0, m1, m38285
+ pmaddwd m0, m158286
+ punpckhwd m1, m38287
+ pmaddwd m1, m158288
+8289
+ movu ym4, [r0 + 2 * r1 + mmsize]8290
+ vinserti32x8 m4, [r6 + mmsize], 18291
+ punpcklwd m2, m3, m48292
+ pmaddwd m2, m158293
+ punpckhwd m3, m48294
+ pmaddwd m3, m158295
+8296
+ movu ym5, [r0 + r7 + mmsize]8297
+ vinserti32x8 m5, [r6 + r1 + mmsize], 18298
+ punpcklwd m6, m4, m58299
+ pmaddwd m6, m168300
+ punpckhwd m4, m58301
+ pmaddwd m4, m168302
+8303
+ paddd m0, m68304
+ paddd m1, m48305
+8306
+ movu ym4, [r6 + mmsize]8307
+ vinserti32x8 m4, [r6 + 2 * r1 + mmsize], 18308
+ punpcklwd m6, m5, m48309
+ pmaddwd m6, m168310
+ punpckhwd m5, m48311
+ pmaddwd m5, m168312
+8313
+ paddd m2, m68314
+ paddd m3, m58315
+8316
+ movu ym11, [r6 + r1 + mmsize]8317
+ vinserti32x8 m11, [r6 + r7 + mmsize], 18318
+ punpcklwd m8, m4, m118319
+ pmaddwd m8, m178320
+ punpckhwd m4, m118321
+ pmaddwd m4, m178322
+8323
+ movu ym12, [r6 + 2 * r1 + mmsize]8324
+ vinserti32x8 m12, [r6 + 4 * r1 + mmsize], 18325
+ punpcklwd m10, m11, m128326
+ pmaddwd m10, m178327
+ punpckhwd m11, m128328
+ pmaddwd m11, m178329
+8330
+ movu ym13, [r6 + r7 + mmsize]8331
+ vinserti32x8 m13, [r4 + r1 + mmsize], 18332
+ punpcklwd m14, m12, m138333
+ pmaddwd m14, m188334
+ punpckhwd m12, m138335
+ pmaddwd m12, m188336
+8337
+ paddd m8, m148338
+ paddd m4, m128339
+ paddd m0, m88340
+ paddd m1, m48341
+8342
+ movu ym12, [r6 + 4 * r1 + mmsize]8343
+ vinserti32x8 m12, [r4 + 2 * r1 + mmsize], 18344
+ punpcklwd m14, m13, m128345
+ pmaddwd m14, m188346
+ punpckhwd m13, m128347
+ pmaddwd m13, m188348
+8349
+ paddd m10, m148350
+ paddd m11, m138351
+ paddd m2, m108352
+ paddd m3, m118353
+8354
+%ifidn %1, sp8355
+ paddd m0, m198356
+ paddd m1, m198357
+ paddd m2, m198358
+ paddd m3, m198359
+8360
+ psrad m0, INTERP_SHIFT_SP8361
+ psrad m1, INTERP_SHIFT_SP8362
+ psrad m2, INTERP_SHIFT_SP8363
+ psrad m3, INTERP_SHIFT_SP8364
+8365
+ packssdw m0, m18366
+ packssdw m2, m38367
+ CLIPW2 m0, m2, m20, m218368
+%else8369
+ psrad m0, 68370
+ psrad m1, 68371
+ psrad m2, 68372
+ psrad m3, 68373
+8374
+ packssdw m0, m18375
+ packssdw m2, m38376
+%endif8377
+8378
+ movu [r2 + mmsize], ym08379
+ movu [r2 + r3 + mmsize], ym28380
+ vextracti32x8 [r2 + 2 * r3 + mmsize], m0, 18381
+ vextracti32x8 [r2 + r5 + mmsize], m2, 18382
+%endmacro8383
+;-----------------------------------------------------------------------------------------------------------------8384
+; void interp_4tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)8385
+;-----------------------------------------------------------------------------------------------------------------8386
+%macro FILTER_VER_S_LUMA_48x64_AVX512 18387
+INIT_ZMM avx5128388
+cglobal interp_8tap_vert_%1_48x64, 5, 8, 228389
+ add r1d, r1d8390
+ add r3d, r3d8391
+ lea r7, [3 * r1]8392
+ sub r0, r78393
+ shl r4d, 88394
+%ifdef PIC8395
+ lea r5, [tab_LumaCoeffVer_avx512]8396
+ mova m15, [r5 + r4]8397
+ mova m16, [r5 + r4 + 1 * mmsize]8398
+ mova m17, [r5 + r4 + 2 * mmsize]8399
+ mova m18, [r5 + r4 + 3 * mmsize]8400
+%else8401
+ lea r5, [tab_LumaCoeffVer_avx512 + r4]8402
+ mova m15, [r5]8403
+ mova m16, [r5 + 1 * mmsize]8404
+ mova m17, [r5 + 2 * mmsize]8405
+ mova m18, [r5 + 3 * mmsize]8406
+%endif8407
+%ifidn %1, sp8408
+ vbroadcasti32x4 m19, [INTERP_OFFSET_SP]8409
+ pxor m20, m208410
+ vbroadcasti32x8 m21, [pw_pixel_max]8411
+%endif8412
+8413
+ lea r5, [3 * r3]8414
+%rep 158415
+ PROCESS_LUMA_VERT_S_48x4_AVX512 %18416
+ lea r0, [r0 + 4 * r1]8417
+ lea r2, [r2 + 4 * r3]8418
+%endrep8419
+ PROCESS_LUMA_VERT_S_48x4_AVX512 %18420
+ RET8421
+%endmacro8422
+8423
+%if ARCH_X86_648424
+ FILTER_VER_S_LUMA_48x64_AVX512 ss8425
+ FILTER_VER_S_LUMA_48x64_AVX512 sp8426
+%endif8427
+8428
+%macro PROCESS_LUMA_VERT_S_64x2_AVX512 18429
+ movu m1, [r0] ;0 row8430
+ movu m3, [r0 + r1] ;1 row8431
+ punpcklwd m0, m1, m38432
+ pmaddwd m0, m158433
+ punpckhwd m1, m38434
+ pmaddwd m1, m158435
+8436
+ movu m4, [r0 + 2 * r1] ;2 row8437
+ punpcklwd m2, m3, m48438
+ pmaddwd m2, m158439
+ punpckhwd m3, m48440
+ pmaddwd m3, m158441
+8442
+ movu m5, [r0 + r7] ;3 row8443
+ punpcklwd m6, m4, m58444
+ pmaddwd m6, m168445
+ punpckhwd m4, m58446
+ pmaddwd m4, m168447
+8448
+ paddd m0, m68449
+ paddd m1, m48450
+8451
+ movu m4, [r0 + 4 * r1] ;4 row8452
+ punpcklwd m6, m5, m48453
+ pmaddwd m6, m168454
+ punpckhwd m5, m48455
+ pmaddwd m5, m168456
+8457
+ paddd m2, m68458
+ paddd m3, m58459
+8460
+ lea r6, [r0 + 4 * r1]8461
+8462
+ movu m11, [r6 + r1] ;5 row8463
+ punpcklwd m8, m4, m118464
+ pmaddwd m8, m178465
+ punpckhwd m4, m118466
+ pmaddwd m4, m178467
+8468
+ movu m12, [r6 + 2 * r1] ;6 row8469
+ punpcklwd m10, m11, m128470
+ pmaddwd m10, m178471
+ punpckhwd m11, m128472
+ pmaddwd m11, m178473
+8474
+ movu m13, [r6 + r7] ;7 row8475
+ punpcklwd m14, m12, m138476
+ pmaddwd m14, m188477
+ punpckhwd m12, m138478
+ pmaddwd m12, m188479
+8480
+ paddd m8, m148481
+ paddd m4, m128482
+ paddd m0, m88483
+ paddd m1, m48484
+8485
+ movu m12, [r6 + 4 * r1] ; 8 row8486
+ punpcklwd m14, m13, m128487
+ pmaddwd m14, m188488
+ punpckhwd m13, m128489
+ pmaddwd m13, m188490
+8491
+ paddd m10, m148492
+ paddd m11, m138493
+ paddd m2, m108494
+ paddd m3, m118495
+8496
+%ifidn %1, sp8497
+ paddd m0, m198498
+ paddd m1, m198499
+ paddd m2, m198500
+ paddd m3, m198501
+8502
+ psrad m0, INTERP_SHIFT_SP8503
+ psrad m1, INTERP_SHIFT_SP8504
+ psrad m2, INTERP_SHIFT_SP8505
+ psrad m3, INTERP_SHIFT_SP8506
+8507
+ packssdw m0, m18508
+ packssdw m2, m38509
+ CLIPW2 m0, m2, m20, m218510
+%else8511
+ psrad m0, 68512
+ psrad m1, 68513
+ psrad m2, 68514
+ psrad m3, 68515
+8516
+ packssdw m0, m18517
+ packssdw m2, m38518
+%endif8519
+8520
+ movu [r2], m08521
+ movu [r2 + r3], m28522
+8523
+ movu m1, [r0 + mmsize] ;0 row8524
+ movu m3, [r0 + r1 + mmsize] ;1 row8525
+ punpcklwd m0, m1, m38526
+ pmaddwd m0, m158527
+ punpckhwd m1, m38528
+ pmaddwd m1, m158529
+8530
+ movu m4, [r0 + 2 * r1 + mmsize] ;2 row8531
+ punpcklwd m2, m3, m48532
+ pmaddwd m2, m158533
+ punpckhwd m3, m48534
+ pmaddwd m3, m158535
+8536
+ movu m5, [r0 + r7 + mmsize] ;3 row8537
+ punpcklwd m6, m4, m58538
+ pmaddwd m6, m168539
+ punpckhwd m4, m58540
+ pmaddwd m4, m168541
+8542
+ paddd m0, m68543
+ paddd m1, m48544
+8545
+ movu m4, [r0 + 4 * r1 + mmsize] ;4 row8546
+ punpcklwd m6, m5, m48547
+ pmaddwd m6, m168548
+ punpckhwd m5, m48549
+ pmaddwd m5, m168550
+8551
+ paddd m2, m68552
+ paddd m3, m58553
+8554
+ movu m11, [r6 + r1 + mmsize] ;5 row8555
+ punpcklwd m8, m4, m118556
+ pmaddwd m8, m178557
+ punpckhwd m4, m118558
+ pmaddwd m4, m178559
+8560
+ movu m12, [r6 + 2 * r1 + mmsize] ;6 row8561
+ punpcklwd m10, m11, m128562
+ pmaddwd m10, m178563
+ punpckhwd m11, m128564
+ pmaddwd m11, m178565
+8566
+ movu m13, [r6 + r7 + mmsize] ;7 row8567
+ punpcklwd m14, m12, m138568
+ pmaddwd m14, m188569
+ punpckhwd m12, m138570
+ pmaddwd m12, m188571
+8572
+ paddd m8, m148573
+ paddd m4, m128574
+ paddd m0, m88575
+ paddd m1, m48576
+8577
+ movu m12, [r6 + 4 * r1 + mmsize] ; 8 row8578
+ punpcklwd m14, m13, m128579
+ pmaddwd m14, m188580
+ punpckhwd m13, m128581
+ pmaddwd m13, m188582
+8583
+ paddd m10, m148584
+ paddd m11, m138585
+ paddd m2, m108586
+ paddd m3, m118587
+8588
+%ifidn %1, sp8589
+ paddd m0, m198590
+ paddd m1, m198591
+ paddd m2, m198592
+ paddd m3, m198593
+8594
+ psrad m0, INTERP_SHIFT_SP8595
+ psrad m1, INTERP_SHIFT_SP8596
+ psrad m2, INTERP_SHIFT_SP8597
+ psrad m3, INTERP_SHIFT_SP8598
+8599
+ packssdw m0, m18600
+ packssdw m2, m38601
+ CLIPW2 m0, m2, m20, m218602
+%else8603
+ psrad m0, 68604
+ psrad m1, 68605
+ psrad m2, 68606
+ psrad m3, 68607
+8608
+ packssdw m0, m18609
+ packssdw m2, m38610
+%endif8611
+8612
+ movu [r2 + mmsize], m08613
+ movu [r2 + r3 + mmsize], m28614
+%endmacro8615
+;-----------------------------------------------------------------------------------------------------------------8616
+; void interp_8tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)8617
+;-----------------------------------------------------------------------------------------------------------------8618
+%macro FILTER_VER_S_LUMA_64xN_AVX512 28619
+INIT_ZMM avx5128620
+cglobal interp_8tap_vert_%1_64x%2, 5, 8, 228621
+ add r1d, r1d8622
+ add r3d, r3d8623
+ lea r7, [3 * r1]8624
+ sub r0, r78625
+ shl r4d, 88626
+%ifdef PIC8627
+ lea r5, [tab_LumaCoeffVer_avx512]8628
+ mova m15, [r5 + r4]8629
+ mova m16, [r5 + r4 + 1 * mmsize]8630
+ mova m17, [r5 + r4 + 2 * mmsize]8631
+ mova m18, [r5 + r4 + 3 * mmsize]8632
+%else8633
+ lea r5, [tab_LumaCoeffVer_avx512 + r4]8634
+ mova m15, [r5]8635
+ mova m16, [r5 + 1 * mmsize]8636
+ mova m17, [r5 + 2 * mmsize]8637
+ mova m18, [r5 + 3 * mmsize]8638
+%endif8639
+%ifidn %1, sp8640
+ vbroadcasti32x4 m19, [INTERP_OFFSET_SP]8641
+ pxor m20, m208642
+ vbroadcasti32x8 m21, [pw_pixel_max]8643
+%endif8644
+8645
+%rep %2/2 - 18646
+ PROCESS_LUMA_VERT_S_64x2_AVX512 %18647
+ lea r0, [r0 + 2 * r1]8648
+ lea r2, [r2 + 2 * r3]8649
+%endrep8650
+ PROCESS_LUMA_VERT_S_64x2_AVX512 %18651
+ RET8652
+%endmacro8653
+8654
+%if ARCH_X86_648655
+ FILTER_VER_S_LUMA_64xN_AVX512 ss, 168656
+ FILTER_VER_S_LUMA_64xN_AVX512 ss, 328657
+ FILTER_VER_S_LUMA_64xN_AVX512 ss, 488658
+ FILTER_VER_S_LUMA_64xN_AVX512 ss, 648659
+ FILTER_VER_S_LUMA_64xN_AVX512 sp, 168660
+ FILTER_VER_S_LUMA_64xN_AVX512 sp, 328661
+ FILTER_VER_S_LUMA_64xN_AVX512 sp, 488662
+ FILTER_VER_S_LUMA_64xN_AVX512 sp, 648663
+%endif8664
+;-------------------------------------------------------------------------------------------------------------8665
+;avx512 luma_vss and luma_vsp code end8666
+;-------------------------------------------------------------------------------------------------------------8667
+;-------------------------------------------------------------------------------------------------------------8668
+;avx512 luma_vpp and luma_vps code start8669
+;-------------------------------------------------------------------------------------------------------------8670
+%macro PROCESS_LUMA_VERT_P_16x4_AVX512 18671
+ lea r5, [r0 + 4 * r1]8672
+ movu ym1, [r0]8673
+ movu ym3, [r0 + r1]8674
+ vinserti32x8 m1, [r0 + 2 * r1], 18675
+ vinserti32x8 m3, [r0 + r7], 18676
+ punpcklwd m0, m1, m38677
+ pmaddwd m0, m158678
+ punpckhwd m1, m38679
+ pmaddwd m1, m158680
+8681
+ movu ym4, [r0 + 2 * r1]8682
+ vinserti32x8 m4, [r0 + 4 * r1], 18683
+ punpcklwd m2, m3, m48684
+ pmaddwd m2, m158685
+ punpckhwd m3, m48686
+ pmaddwd m3, m158687
+8688
+ movu ym5, [r0 + r7]8689
+ vinserti32x8 m5, [r5 + r1], 18690
+ punpcklwd m6, m4, m58691
+ pmaddwd m6, m168692
+ punpckhwd m4, m58693
+ pmaddwd m4, m168694
+8695
+ paddd m0, m68696
+ paddd m1, m48697
+8698
+ movu ym4, [r5]8699
+ vinserti32x8 m4, [r5 + 2 * r1], 18700
+ punpcklwd m6, m5, m48701
+ pmaddwd m6, m168702
+ punpckhwd m5, m48703
+ pmaddwd m5, m168704
+8705
+ paddd m2, m68706
+ paddd m3, m58707
+8708
+ lea r4, [r5 + 4 * r1]8709
+ movu ym11, [r5 + r1]8710
+ vinserti32x8 m11, [r5 + r7], 18711
+ punpcklwd m8, m4, m118712
+ pmaddwd m8, m178713
+ punpckhwd m4, m118714
+ pmaddwd m4, m178715
+8716
+ movu ym12, [r5 + 2 * r1]8717
+ vinserti32x8 m12, [r4], 18718
+ punpcklwd m10, m11, m128719
+ pmaddwd m10, m178720
+ punpckhwd m11, m128721
+ pmaddwd m11, m178722
+8723
+ movu ym13, [r5 + r7]8724
+ vinserti32x8 m13, [r4 + r1], 18725
+ punpcklwd m14, m12, m138726
+ pmaddwd m14, m188727
+ punpckhwd m12, m138728
+ pmaddwd m12, m188729
+8730
+ paddd m8, m148731
+ paddd m4, m128732
+ paddd m0, m88733
+ paddd m1, m48734
+8735
+ movu ym12, [r4]8736
+ vinserti32x8 m12, [r4 + 2 * r1], 18737
+ punpcklwd m14, m13, m128738
+ pmaddwd m14, m188739
+ punpckhwd m13, m128740
+ pmaddwd m13, m188741
+8742
+ paddd m10, m148743
+ paddd m11, m138744
+ paddd m2, m108745
+ paddd m3, m118746
+8747
+ paddd m0, m198748
+ paddd m1, m198749
+ paddd m2, m198750
+ paddd m3, m198751
+8752
+%ifidn %1, pp8753
+ psrad m0, INTERP_SHIFT_PP8754
+ psrad m1, INTERP_SHIFT_PP8755
+ psrad m2, INTERP_SHIFT_PP8756
+ psrad m3, INTERP_SHIFT_PP8757
+8758
+ packssdw m0, m18759
+ packssdw m2, m38760
+ CLIPW2 m0, m2, m20, m218761
+%else8762
+ psrad m0, INTERP_SHIFT_PS8763
+ psrad m1, INTERP_SHIFT_PS8764
+ psrad m2, INTERP_SHIFT_PS8765
+ psrad m3, INTERP_SHIFT_PS8766
+8767
+ packssdw m0, m18768
+ packssdw m2, m38769
+%endif8770
+8771
+ movu [r2], ym08772
+ movu [r2 + r3], ym28773
+ vextracti32x8 [r2 + 2 * r3], m0, 18774
+ vextracti32x8 [r2 + r8], m2, 18775
+%endmacro8776
+;-----------------------------------------------------------------------------------------------------------------8777
+; void interp_4tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)8778
+;-----------------------------------------------------------------------------------------------------------------8779
+%macro FILTER_VER_P_LUMA_16xN_AVX512 28780
+INIT_ZMM avx5128781
+cglobal interp_8tap_vert_%1_16x%2, 5, 9, 228782
+ add r1d, r1d8783
+ add r3d, r3d8784
+ shl r4d, 88785
+%ifdef PIC8786
+ lea r5, [tab_LumaCoeffVer_avx512]8787
+ mova m15, [r5 + r4]8788
+ mova m16, [r5 + r4 + 1 * mmsize]8789
+ mova m17, [r5 + r4 + 2 * mmsize]8790
+ mova m18, [r5 + r4 + 3 * mmsize]8791
+%else8792
+ lea r5, [tab_LumaCoeffVer_avx512 + r4]8793
+ mova m15, [r5]8794
+ mova m16, [r5 + 1 * mmsize]8795
+ mova m17, [r5 + 2 * mmsize]8796
+ mova m18, [r5 + 3 * mmsize]8797
+%endif8798
+%ifidn %1, pp8799
+ vbroadcasti32x4 m19, [INTERP_OFFSET_PP]8800
+ pxor m20, m208801
+ vbroadcasti32x8 m21, [pw_pixel_max]8802
+%else8803
+ vbroadcasti32x4 m19, [INTERP_OFFSET_PS]8804
+%endif8805
+ lea r7, [3 * r1]8806
+ lea r8, [3 * r3]8807
+ sub r0, r78808
+8809
+%rep %2/4 - 18810
+ PROCESS_LUMA_VERT_P_16x4_AVX512 %18811
+ lea r0, [r0 + 4 * r1]8812
+ lea r2, [r2 + 4 * r3]8813
+%endrep8814
+ PROCESS_LUMA_VERT_P_16x4_AVX512 %18815
+ RET8816
+%endmacro8817
+8818
+%if ARCH_X86_648819
+ FILTER_VER_P_LUMA_16xN_AVX512 ps, 48820
+ FILTER_VER_P_LUMA_16xN_AVX512 ps, 88821
+ FILTER_VER_P_LUMA_16xN_AVX512 ps, 128822
+ FILTER_VER_P_LUMA_16xN_AVX512 ps, 168823
+ FILTER_VER_P_LUMA_16xN_AVX512 ps, 328824
+ FILTER_VER_P_LUMA_16xN_AVX512 ps, 648825
+ FILTER_VER_P_LUMA_16xN_AVX512 pp, 48826
+ FILTER_VER_P_LUMA_16xN_AVX512 pp, 88827
+ FILTER_VER_P_LUMA_16xN_AVX512 pp, 128828
+ FILTER_VER_P_LUMA_16xN_AVX512 pp, 168829
+ FILTER_VER_P_LUMA_16xN_AVX512 pp, 328830
+ FILTER_VER_P_LUMA_16xN_AVX512 pp, 648831
+%endif8832
+8833
+%macro PROCESS_LUMA_VERT_P_24x4_AVX512 18834
+ PROCESS_LUMA_VERT_P_16x4_AVX512 %18835
+ movu xm1, [r0 + mmsize/2]8836
+ movu xm3, [r0 + r1 + mmsize/2]8837
+ vinserti32x4 m1, [r0 + r1 + mmsize/2], 18838
+ vinserti32x4 m3, [r0 + 2 * r1 + mmsize/2], 18839
+ vinserti32x4 m1, [r0 + 2 * r1 + mmsize/2], 28840
+ vinserti32x4 m3, [r0 + r7 + mmsize/2], 28841
+ vinserti32x4 m1, [r0 + r7 + mmsize/2], 38842
+ vinserti32x4 m3, [r0 + 4 * r1 + mmsize/2], 38843
+8844
+ punpcklwd m0, m1, m38845
+ pmaddwd m0, m158846
+ punpckhwd m1, m38847
+ pmaddwd m1, m158848
+8849
+ movu xm4, [r0 + 2 * r1 + mmsize/2]8850
+ movu xm5, [r0 + r7 + mmsize/2]8851
+ vinserti32x4 m4, [r0 + r7 + mmsize/2], 18852
+ vinserti32x4 m5, [r5 + mmsize/2], 18853
+ vinserti32x4 m4, [r5 + mmsize/2], 28854
+ vinserti32x4 m5, [r5 + r1 + mmsize/2], 28855
+ vinserti32x4 m4, [r5 + r1 + mmsize/2], 38856
+ vinserti32x4 m5, [r5 + 2 * r1 + mmsize/2], 38857
+8858
+ punpcklwd m3, m4, m58859
+ pmaddwd m3, m168860
+ punpckhwd m4, m58861
+ pmaddwd m4, m168862
+8863
+ paddd m0, m38864
+ paddd m1, m48865
+8866
+ movu xm3, [r5 + mmsize/2]8867
+ movu xm5, [r5 + r1 + mmsize/2]8868
+ vinserti32x4 m3, [r5 + r1 + mmsize/2], 18869
+ vinserti32x4 m5, [r5 + 2 * r1 + mmsize/2], 18870
+ vinserti32x4 m3, [r5 + 2 * r1 + mmsize/2], 28871
+ vinserti32x4 m5, [r5 + r7 + mmsize/2], 28872
+ vinserti32x4 m3, [r5 + r7 + mmsize/2], 38873
+ vinserti32x4 m5, [r5 + 4 * r1 + mmsize/2], 38874
+8875
+ punpcklwd m2, m3, m58876
+ pmaddwd m2, m178877
+ punpckhwd m3, m58878
+ pmaddwd m3, m178879
+8880
+ movu xm6, [r5 + 2 * r1 + mmsize/2]8881
+ movu xm7, [r5 + r7 + mmsize/2]8882
+ vinserti32x4 m6, [r5 + r7 + mmsize/2], 18883
+ vinserti32x4 m7, [r4 + mmsize/2], 18884
+ vinserti32x4 m6, [r4 + mmsize/2], 28885
+ vinserti32x4 m7, [r4 + r1 + mmsize/2], 28886
+ vinserti32x4 m6, [r4 + r1 + mmsize/2], 38887
+ vinserti32x4 m7, [r4 + 2 * r1 + mmsize/2], 38888
+8889
+ punpcklwd m5, m6, m78890
+ pmaddwd m5, m188891
+ punpckhwd m6, m78892
+ pmaddwd m6, m188893
+8894
+ paddd m2, m58895
+ paddd m3, m68896
+ paddd m0, m28897
+ paddd m1, m38898
+8899
+ paddd m0, m198900
+ paddd m1, m198901
+8902
+%ifidn %1, pp8903
+ psrad m0, INTERP_SHIFT_PP8904
+ psrad m1, INTERP_SHIFT_PP8905
+ packssdw m0, m18906
+ CLIPW m0, m20, m218907
+%else8908
+ psrad m0, INTERP_SHIFT_PS8909
+ psrad m1, INTERP_SHIFT_PS8910
+ packssdw m0, m18911
+%endif8912
+8913
+ movu [r2 + mmsize/2], xm08914
+ vextracti32x4 [r2 + r3 + mmsize/2], m0, 18915
+ vextracti32x4 [r2 + 2 * r3 + mmsize/2], m0, 28916
+ vextracti32x4 [r2 + r8 + mmsize/2], m0, 38917
+%endmacro8918
+;-----------------------------------------------------------------------------------------------------------------8919
+; void interp_4tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)8920
+;-----------------------------------------------------------------------------------------------------------------8921
+%macro FILTER_VER_P_LUMA_24xN_AVX512 18922
+INIT_ZMM avx5128923
+cglobal interp_8tap_vert_%1_24x32, 5, 9, 228924
+ add r1d, r1d8925
+ add r3d, r3d8926
+ shl r4d, 88927
+%ifdef PIC8928
+ lea r5, [tab_LumaCoeffVer_avx512]8929
+ mova m15, [r5 + r4]8930
+ mova m16, [r5 + r4 + 1 * mmsize]8931
+ mova m17, [r5 + r4 + 2 * mmsize]8932
+ mova m18, [r5 + r4 + 3 * mmsize]8933
+%else8934
+ lea r5, [tab_LumaCoeffVer_avx512 + r4]8935
+ mova m15, [r5]8936
+ mova m16, [r5 + 1 * mmsize]8937
+ mova m17, [r5 + 2 * mmsize]8938
+ mova m18, [r5 + 3 * mmsize]8939
+%endif8940
+%ifidn %1, pp8941
+ vbroadcasti32x4 m19, [INTERP_OFFSET_PP]8942
+ pxor m20, m208943
+ vbroadcasti32x8 m21, [pw_pixel_max]8944
+%else8945
+ vbroadcasti32x4 m19, [INTERP_OFFSET_PS]8946
+%endif8947
+ lea r7, [3 * r1]8948
+ lea r8, [3 * r3]8949
+ sub r0, r78950
+8951
+%rep 78952
+ PROCESS_LUMA_VERT_P_24x4_AVX512 %18953
+ lea r0, [r0 + 4 * r1]8954
+ lea r2, [r2 + 4 * r3]8955
+%endrep8956
+ PROCESS_LUMA_VERT_P_24x4_AVX512 %18957
+ RET8958
+%endmacro8959
+8960
+%if ARCH_X86_648961
+ FILTER_VER_P_LUMA_24xN_AVX512 ps8962
+ FILTER_VER_P_LUMA_24xN_AVX512 pp8963
+%endif8964
+8965
+%macro PROCESS_LUMA_VERT_P_32x2_AVX512 18966
+ movu m1, [r0] ;0 row8967
+ movu m3, [r0 + r1] ;1 row8968
+ punpcklwd m0, m1, m38969
+ pmaddwd m0, m158970
+ punpckhwd m1, m38971
+ pmaddwd m1, m158972
+8973
+ movu m4, [r0 + 2 * r1] ;2 row8974
+ punpcklwd m2, m3, m48975
+ pmaddwd m2, m158976
+ punpckhwd m3, m48977
+ pmaddwd m3, m158978
+8979
+ movu m5, [r0 + r7] ;3 row8980
+ punpcklwd m6, m4, m58981
+ pmaddwd m6, m168982
+ punpckhwd m4, m58983
+ pmaddwd m4, m168984
+8985
+ paddd m0, m68986
+ paddd m1, m48987
+8988
+ movu m4, [r0 + 4 * r1] ;4 row8989
+ punpcklwd m6, m5, m48990
+ pmaddwd m6, m168991
+ punpckhwd m5, m48992
+ pmaddwd m5, m168993
+8994
+ paddd m2, m68995
+ paddd m3, m58996
+8997
+ lea r6, [r0 + 4 * r1]8998
+8999
+ movu m11, [r6 + r1] ;5 row9000
+ punpcklwd m8, m4, m119001
+ pmaddwd m8, m179002
+ punpckhwd m4, m119003
+ pmaddwd m4, m179004
+9005
+ movu m12, [r6 + 2 * r1] ;6 row9006
+ punpcklwd m10, m11, m129007
+ pmaddwd m10, m179008
+ punpckhwd m11, m129009
+ pmaddwd m11, m179010
+9011
+ movu m13, [r6 + r7] ;7 row9012
+ punpcklwd m14, m12, m139013
+ pmaddwd m14, m189014
+ punpckhwd m12, m139015
+ pmaddwd m12, m189016
+9017
+ paddd m8, m149018
+ paddd m4, m129019
+ paddd m0, m89020
+ paddd m1, m49021
+9022
+ movu m12, [r6 + 4 * r1] ; 8 row9023
+ punpcklwd m14, m13, m129024
+ pmaddwd m14, m189025
+ punpckhwd m13, m129026
+ pmaddwd m13, m189027
+9028
+ paddd m10, m149029
+ paddd m11, m139030
+ paddd m2, m109031
+ paddd m3, m119032
+9033
+ paddd m0, m199034
+ paddd m1, m199035
+ paddd m2, m199036
+ paddd m3, m199037
+9038
+%ifidn %1, pp9039
+ psrad m0, INTERP_SHIFT_PP9040
+ psrad m1, INTERP_SHIFT_PP9041
+ psrad m2, INTERP_SHIFT_PP9042
+ psrad m3, INTERP_SHIFT_PP9043
+9044
+ packssdw m0, m19045
+ packssdw m2, m39046
+ CLIPW2 m0, m2, m20, m219047
+%else9048
+ psrad m0, INTERP_SHIFT_PS9049
+ psrad m1, INTERP_SHIFT_PS9050
+ psrad m2, INTERP_SHIFT_PS9051
+ psrad m3, INTERP_SHIFT_PS9052
+9053
+ packssdw m0, m19054
+ packssdw m2, m39055
+%endif9056
+9057
+ movu [r2], m09058
+ movu [r2 + r3], m29059
+%endmacro9060
+;-----------------------------------------------------------------------------------------------------------------9061
+; void interp_4tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)9062
+;-----------------------------------------------------------------------------------------------------------------9063
+%macro FILTER_VER_P_LUMA_32xN_AVX512 29064
+INIT_ZMM avx5129065
+cglobal interp_8tap_vert_%1_32x%2, 5, 8, 229066
+ add r1d, r1d9067
+ add r3d, r3d9068
+ shl r4d, 89069
+%ifdef PIC9070
+ lea r5, [tab_LumaCoeffVer_avx512]9071
+ mova m15, [r5 + r4]9072
+ mova m16, [r5 + r4 + 1 * mmsize]9073
+ mova m17, [r5 + r4 + 2 * mmsize]9074
+ mova m18, [r5 + r4 + 3 * mmsize]9075
+%else9076
+ lea r5, [tab_LumaCoeffVer_avx512 + r4]9077
+ mova m15, [r5]9078
+ mova m16, [r5 + 1 * mmsize]9079
+ mova m17, [r5 + 2 * mmsize]9080
+ mova m18, [r5 + 3 * mmsize]9081
+%endif9082
+%ifidn %1, pp9083
+ vbroadcasti32x4 m19, [INTERP_OFFSET_PP]9084
+ pxor m20, m209085
+ vbroadcasti32x8 m21, [pw_pixel_max]9086
+%else9087
+ vbroadcasti32x4 m19, [INTERP_OFFSET_PS]9088
+%endif9089
+ lea r7, [3 * r1]9090
+ sub r0, r79091
+9092
+%rep %2/2 - 19093
+ PROCESS_LUMA_VERT_P_32x2_AVX512 %19094
+ lea r0, [r0 + 2 * r1]9095
+ lea r2, [r2 + 2 * r3]9096
+%endrep9097
+ PROCESS_LUMA_VERT_P_32x2_AVX512 %19098
+ RET9099
+%endmacro9100
+9101
+%if ARCH_X86_649102
+ FILTER_VER_P_LUMA_32xN_AVX512 ps, 89103
+ FILTER_VER_P_LUMA_32xN_AVX512 ps, 169104
+ FILTER_VER_P_LUMA_32xN_AVX512 ps, 329105
+ FILTER_VER_P_LUMA_32xN_AVX512 ps, 249106
+ FILTER_VER_P_LUMA_32xN_AVX512 ps, 649107
+ FILTER_VER_P_LUMA_32xN_AVX512 pp, 89108
+ FILTER_VER_P_LUMA_32xN_AVX512 pp, 169109
+ FILTER_VER_P_LUMA_32xN_AVX512 pp, 329110
+ FILTER_VER_P_LUMA_32xN_AVX512 pp, 249111
+ FILTER_VER_P_LUMA_32xN_AVX512 pp, 649112
+%endif9113
+9114
+%macro PROCESS_LUMA_VERT_P_48x4_AVX512 19115
+ PROCESS_LUMA_VERT_P_32x2_AVX512 %19116
+ movu m1, [r0 + 2 * r1]9117
+ movu m3, [r0 + r7]9118
+ punpcklwd m0, m1, m39119
+ pmaddwd m0, m159120
+ punpckhwd m1, m39121
+ pmaddwd m1, m159122
+9123
+ movu m4, [r0 + 4 * r1]9124
+ punpcklwd m2, m3, m49125
+ pmaddwd m2, m159126
+ punpckhwd m3, m49127
+ pmaddwd m3, m159128
+9129
+ movu m5, [r6 + r1]9130
+ punpcklwd m6, m4, m59131
+ pmaddwd m6, m169132
+ punpckhwd m4, m59133
+ pmaddwd m4, m169134
+9135
+ paddd m0, m69136
+ paddd m1, m49137
+9138
+ movu m4, [r6 + 2 * r1]9139
+ punpcklwd m6, m5, m49140
+ pmaddwd m6, m169141
+ punpckhwd m5, m49142
+ pmaddwd m5, m169143
+9144
+ paddd m2, m69145
+ paddd m3, m59146
+9147
+ lea r4, [r6 + 4 * r1]9148
+9149
+ movu m11, [r6 + r7]9150
+ punpcklwd m8, m4, m119151
+ pmaddwd m8, m179152
+ punpckhwd m4, m119153
+ pmaddwd m4, m179154
+9155
+ movu m12, [r6 + 4 * r1]9156
+ punpcklwd m10, m11, m129157
+ pmaddwd m10, m179158
+ punpckhwd m11, m129159
+ pmaddwd m11, m179160
+9161
+ movu m13, [r4 + r1]9162
+ punpcklwd m14, m12, m139163
+ pmaddwd m14, m189164
+ punpckhwd m12, m139165
+ pmaddwd m12, m189166
+9167
+ paddd m8, m149168
+ paddd m4, m129169
+ paddd m0, m89170
+ paddd m1, m49171
+9172
+ movu m12, [r4 + 2 * r1]9173
+ punpcklwd m14, m13, m129174
+ pmaddwd m14, m189175
+ punpckhwd m13, m129176
+ pmaddwd m13, m189177
+9178
+ paddd m10, m149179
+ paddd m11, m139180
+ paddd m2, m109181
+ paddd m3, m119182
+9183
+ paddd m0, m199184
+ paddd m1, m199185
+ paddd m2, m199186
+ paddd m3, m199187
+9188
+%ifidn %1, pp9189
+ psrad m0, INTERP_SHIFT_PP9190
+ psrad m1, INTERP_SHIFT_PP9191
+ psrad m2, INTERP_SHIFT_PP9192
+ psrad m3, INTERP_SHIFT_PP9193
+9194
+ packssdw m0, m19195
+ packssdw m2, m39196
+ CLIPW2 m0, m2, m20, m219197
+%else9198
+ psrad m0, INTERP_SHIFT_PS9199
+ psrad m1, INTERP_SHIFT_PS9200
+ psrad m2, INTERP_SHIFT_PS9201
+ psrad m3, INTERP_SHIFT_PS9202
+9203
+ packssdw m0, m19204
+ packssdw m2, m39205
+%endif9206
+ movu [r2 + 2 * r3], m09207
+ movu [r2 + r8], m29208
+9209
+ movu ym1, [r0 + mmsize]9210
+ movu ym3, [r0 + r1 + mmsize]9211
+ vinserti32x8 m1, [r0 + 2 * r1 + mmsize], 19212
+ vinserti32x8 m3, [r0 + r7 + mmsize], 19213
+ punpcklwd m0, m1, m39214
+ pmaddwd m0, m159215
+ punpckhwd m1, m39216
+ pmaddwd m1, m159217
+9218
+ movu ym4, [r0 + 2 * r1 + mmsize]9219
+ vinserti32x8 m4, [r0 + 4 * r1 + mmsize], 19220
+ punpcklwd m2, m3, m49221
+ pmaddwd m2, m159222
+ punpckhwd m3, m49223
+ pmaddwd m3, m159224
+9225
+ movu ym5, [r0 + r7 + mmsize]9226
+ vinserti32x8 m5, [r6 + r1 + mmsize], 19227
+ punpcklwd m6, m4, m59228
+ pmaddwd m6, m169229
+ punpckhwd m4, m59230
+ pmaddwd m4, m169231
+9232
+ paddd m0, m69233
+ paddd m1, m49234
+9235
+ movu ym4, [r6 + mmsize]9236
+ vinserti32x8 m4, [r6 + 2 * r1 + mmsize], 19237
+ punpcklwd m6, m5, m49238
+ pmaddwd m6, m169239
+ punpckhwd m5, m49240
+ pmaddwd m5, m169241
+9242
+ paddd m2, m69243
+ paddd m3, m59244
+9245
+ movu ym11, [r6 + r1 + mmsize]9246
+ vinserti32x8 m11, [r6 + r7 + mmsize], 19247
+ punpcklwd m8, m4, m119248
+ pmaddwd m8, m179249
+ punpckhwd m4, m119250
+ pmaddwd m4, m179251
+9252
+ movu ym12, [r6 + 2 * r1 + mmsize]9253
+ vinserti32x8 m12, [r4 + mmsize], 19254
+ punpcklwd m10, m11, m129255
+ pmaddwd m10, m179256
+ punpckhwd m11, m129257
+ pmaddwd m11, m179258
+9259
+ movu ym13, [r6 + r7 + mmsize]9260
+ vinserti32x8 m13, [r4 + r1 + mmsize], 19261
+ punpcklwd m14, m12, m139262
+ pmaddwd m14, m189263
+ punpckhwd m12, m139264
+ pmaddwd m12, m189265
+9266
+ paddd m8, m149267
+ paddd m4, m129268
+ paddd m0, m89269
+ paddd m1, m49270
+9271
+ movu ym12, [r4 + mmsize]9272
+ vinserti32x8 m12, [r4 + 2 * r1 + mmsize], 19273
+ punpcklwd m14, m13, m129274
+ pmaddwd m14, m189275
+ punpckhwd m13, m129276
+ pmaddwd m13, m189277
+9278
+ paddd m10, m149279
+ paddd m11, m139280
+ paddd m2, m109281
+ paddd m3, m119282
+9283
+ paddd m0, m199284
+ paddd m1, m199285
+ paddd m2, m199286
+ paddd m3, m199287
+9288
+%ifidn %1, pp9289
+ psrad m0, INTERP_SHIFT_PP9290
+ psrad m1, INTERP_SHIFT_PP9291
+ psrad m2, INTERP_SHIFT_PP9292
+ psrad m3, INTERP_SHIFT_PP9293
+9294
+ packssdw m0, m19295
+ packssdw m2, m39296
+ CLIPW2 m0, m2, m20, m219297
+%else9298
+ psrad m0, INTERP_SHIFT_PS9299
+ psrad m1, INTERP_SHIFT_PS9300
+ psrad m2, INTERP_SHIFT_PS9301
+ psrad m3, INTERP_SHIFT_PS9302
+9303
+ packssdw m0, m19304
+ packssdw m2, m39305
+%endif9306
+9307
+ movu [r2 + mmsize], ym09308
+ movu [r2 + r3 + mmsize], ym29309
+ vextracti32x8 [r2 + 2 * r3 + mmsize], m0, 19310
+ vextracti32x8 [r2 + r8 + mmsize], m2, 19311
+%endmacro9312
+;-----------------------------------------------------------------------------------------------------------------9313
+; void interp_4tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)9314
+;-----------------------------------------------------------------------------------------------------------------9315
+%macro FILTER_VER_P_LUMA_48x64_AVX512 19316
+INIT_ZMM avx5129317
+cglobal interp_8tap_vert_%1_48x64, 5, 9, 229318
+ add r1d, r1d9319
+ add r3d, r3d9320
+ shl r4d, 89321
+%ifdef PIC9322
+ lea r5, [tab_LumaCoeffVer_avx512]9323
+ mova m15, [r5 + r4]9324
+ mova m16, [r5 + r4 + 1 * mmsize]9325
+ mova m17, [r5 + r4 + 2 * mmsize]9326
+ mova m18, [r5 + r4 + 3 * mmsize]9327
+%else9328
+ lea r5, [tab_LumaCoeffVer_avx512 + r4]9329
+ mova m15, [r5]9330
+ mova m16, [r5 + 1 * mmsize]9331
+ mova m17, [r5 + 2 * mmsize]9332
+ mova m18, [r5 + 3 * mmsize]9333
+%endif9334
+%ifidn %1, pp9335
+ vbroadcasti32x4 m19, [INTERP_OFFSET_PP]9336
+ pxor m20, m209337
+ vbroadcasti32x8 m21, [pw_pixel_max]9338
+%else9339
+ vbroadcasti32x4 m19, [INTERP_OFFSET_PS]9340
+%endif9341
+ lea r7, [3 * r1]9342
+ lea r8, [3 * r3]9343
+ sub r0, r79344
+9345
+%rep 159346
+ PROCESS_LUMA_VERT_P_48x4_AVX512 %19347
+ lea r0, [r0 + 4 * r1]9348
+ lea r2, [r2 + 4 * r3]9349
+%endrep9350
+ PROCESS_LUMA_VERT_P_48x4_AVX512 %19351
+ RET9352
+%endmacro9353
+9354
+%if ARCH_X86_649355
+ FILTER_VER_P_LUMA_48x64_AVX512 ps9356
+ FILTER_VER_P_LUMA_48x64_AVX512 pp9357
+%endif9358
+9359
+%macro PROCESS_LUMA_VERT_P_64x2_AVX512 19360
+ PROCESS_LUMA_VERT_P_32x2_AVX512 %19361
+ movu m1, [r0 + mmsize]9362
+ movu m3, [r0 + r1 + mmsize]9363
+ punpcklwd m0, m1, m39364
+ pmaddwd m0, m159365
+ punpckhwd m1, m39366
+ pmaddwd m1, m159367
+9368
+ movu m4, [r0 + 2 * r1 + mmsize]9369
+ punpcklwd m2, m3, m49370
+ pmaddwd m2, m159371
+ punpckhwd m3, m49372
+ pmaddwd m3, m159373
+9374
+ movu m5, [r0 + r7 + mmsize]9375
+ punpcklwd m6, m4, m59376
+ pmaddwd m6, m169377
+ punpckhwd m4, m59378
+ pmaddwd m4, m169379
+9380
+ paddd m0, m69381
+ paddd m1, m49382
+9383
+ movu m4, [r0 + 4 * r1 + mmsize]9384
+ punpcklwd m6, m5, m49385
+ pmaddwd m6, m169386
+ punpckhwd m5, m49387
+ pmaddwd m5, m169388
+9389
+ paddd m2, m69390
+ paddd m3, m59391
+9392
+ movu m11, [r6 + r1 + mmsize]9393
+ punpcklwd m8, m4, m119394
+ pmaddwd m8, m179395
+ punpckhwd m4, m119396
+ pmaddwd m4, m179397
+9398
+ movu m12, [r6 + 2 * r1 + mmsize]9399
+ punpcklwd m10, m11, m129400
+ pmaddwd m10, m179401
+ punpckhwd m11, m129402
+ pmaddwd m11, m179403
+9404
+ movu m13, [r6 + r7 + mmsize]9405
+ punpcklwd m14, m12, m139406
+ pmaddwd m14, m189407
+ punpckhwd m12, m139408
+ pmaddwd m12, m189409
+9410
+ paddd m8, m149411
+ paddd m4, m129412
+ paddd m0, m89413
+ paddd m1, m49414
+9415
+ movu m12, [r6 + 4 * r1 + mmsize]9416
+ punpcklwd m14, m13, m129417
+ pmaddwd m14, m189418
+ punpckhwd m13, m129419
+ pmaddwd m13, m189420
+9421
+ paddd m10, m149422
+ paddd m11, m139423
+ paddd m2, m109424
+ paddd m3, m119425
+9426
+ paddd m0, m199427
+ paddd m1, m199428
+ paddd m2, m199429
+ paddd m3, m199430
+9431
+%ifidn %1, pp9432
+ psrad m0, INTERP_SHIFT_PP9433
+ psrad m1, INTERP_SHIFT_PP9434
+ psrad m2, INTERP_SHIFT_PP9435
+ psrad m3, INTERP_SHIFT_PP9436
+9437
+ packssdw m0, m19438
+ packssdw m2, m39439
+ CLIPW2 m0, m2, m20, m219440
+%else9441
+ psrad m0, INTERP_SHIFT_PS9442
+ psrad m1, INTERP_SHIFT_PS9443
+ psrad m2, INTERP_SHIFT_PS9444
+ psrad m3, INTERP_SHIFT_PS9445
+9446
+ packssdw m0, m19447
+ packssdw m2, m39448
+%endif9449
+9450
+ movu [r2 + mmsize], m09451
+ movu [r2 + r3 + mmsize], m29452
+%endmacro9453
+;-----------------------------------------------------------------------------------------------------------------9454
+; void interp_8tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)9455
+;-----------------------------------------------------------------------------------------------------------------9456
+%macro FILTER_VER_P_LUMA_64xN_AVX512 29457
+INIT_ZMM avx5129458
+cglobal interp_8tap_vert_%1_64x%2, 5, 8, 229459
+ add r1d, r1d9460
+ add r3d, r3d9461
+ shl r4d, 89462
+%ifdef PIC9463
+ lea r5, [tab_LumaCoeffVer_avx512]9464
+ mova m15, [r5 + r4]9465
+ mova m16, [r5 + r4 + 1 * mmsize]9466
+ mova m17, [r5 + r4 + 2 * mmsize]9467
+ mova m18, [r5 + r4 + 3 * mmsize]9468
+%else9469
+ lea r5, [tab_LumaCoeffVer_avx512 + r4]9470
+ mova m15, [r5]9471
+ mova m16, [r5 + 1 * mmsize]9472
+ mova m17, [r5 + 2 * mmsize]9473
+ mova m18, [r5 + 3 * mmsize]9474
+%endif9475
+%ifidn %1, pp9476
+ vbroadcasti32x4 m19, [INTERP_OFFSET_PP]9477
+ pxor m20, m209478
+ vbroadcasti32x8 m21, [pw_pixel_max]9479
+%else9480
+ vbroadcasti32x4 m19, [INTERP_OFFSET_PS]9481
+%endif9482
+ lea r7, [3 * r1]9483
+ sub r0, r79484
+9485
+%rep %2/2 - 19486
+ PROCESS_LUMA_VERT_P_64x2_AVX512 %19487
+ lea r0, [r0 + 2 * r1]9488
+ lea r2, [r2 + 2 * r3]9489
+%endrep9490
+ PROCESS_LUMA_VERT_P_64x2_AVX512 %19491
+ RET9492
+%endmacro9493
+9494
+%if ARCH_X86_649495
+ FILTER_VER_P_LUMA_64xN_AVX512 ps, 169496
+ FILTER_VER_P_LUMA_64xN_AVX512 ps, 329497
+ FILTER_VER_P_LUMA_64xN_AVX512 ps, 489498
+ FILTER_VER_P_LUMA_64xN_AVX512 ps, 649499
+ FILTER_VER_P_LUMA_64xN_AVX512 pp, 169500
+ FILTER_VER_P_LUMA_64xN_AVX512 pp, 329501
+ FILTER_VER_P_LUMA_64xN_AVX512 pp, 489502
+ FILTER_VER_P_LUMA_64xN_AVX512 pp, 649503
+%endif9504
+;-------------------------------------------------------------------------------------------------------------9505
+;avx512 luma_vpp and luma_vps code end9506
+;-------------------------------------------------------------------------------------------------------------9507
+;-------------------------------------------------------------------------------------------------------------9508
+;ipfilter_luma_avx512 code end9509
+;-------------------------------------------------------------------------------------------------------------9510
x265_2.7.tar.gz/source/common/x86/ipfilter8.asm -> x265_2.9.tar.gz/source/common/x86/ipfilter8.asm
Changed
5651
1
2
%include "x86inc.asm"3
%include "x86util.asm"4
5
-SECTION_RODATA 326
+SECTION_RODATA 647
const tab_Tm, db 0, 1, 2, 3, 1, 2, 3, 4, 2, 3, 4, 5, 3, 4, 5, 68
db 4, 5, 6, 7, 5, 6, 7, 8, 6, 7, 8, 9, 7, 8, 9, 109
db 8, 9,10,11, 9,10,11,12,10,11,12,13,11,12,13, 1410
11
12
const pd_526336, times 8 dd 8192*64+204813
14
+const tab_ChromaCoeff, db 0, 64, 0, 015
+ db -2, 58, 10, -216
+ db -4, 54, 16, -217
+ db -6, 46, 28, -418
+ db -4, 36, 36, -419
+ db -4, 28, 46, -620
+ db -2, 16, 54, -421
+ db -2, 10, 58, -222
+23
const tab_LumaCoeff, db 0, 0, 0, 64, 0, 0, 0, 024
db -1, 4, -10, 58, 17, -5, 1, 025
db -1, 4, -11, 40, 40, -11, 4, -126
27
times 16 db 58, -1028
times 16 db 4, -129
30
+ALIGN 6431
+const tab_ChromaCoeffVer_32_avx512, times 32 db 0, 6432
+ times 32 db 0, 033
+34
+ times 32 db -2, 5835
+ times 32 db 10, -236
+37
+ times 32 db -4, 5438
+ times 32 db 16, -239
+40
+ times 32 db -6, 4641
+ times 32 db 28, -442
+43
+ times 32 db -4, 3644
+ times 32 db 36, -445
+46
+ times 32 db -4, 2847
+ times 32 db 46, -648
+49
+ times 32 db -2, 1650
+ times 32 db 54, -451
+52
+ times 32 db -2, 1053
+ times 32 db 58, -254
+55
+ALIGN 6456
+const pw_ChromaCoeffVer_32_avx512, times 16 dw 0, 6457
+ times 16 dw 0, 058
+59
+ times 16 dw -2, 5860
+ times 16 dw 10, -261
+62
+ times 16 dw -4, 5463
+ times 16 dw 16, -264
+65
+ times 16 dw -6, 4666
+ times 16 dw 28, -467
+68
+ times 16 dw -4, 3669
+ times 16 dw 36, -470
+71
+ times 16 dw -4, 2872
+ times 16 dw 46, -673
+74
+ times 16 dw -2, 1675
+ times 16 dw 54, -476
+77
+ times 16 dw -2, 1078
+ times 16 dw 58, -279
+80
+ALIGN 6481
+const pw_LumaCoeffVer_avx512, times 16 dw 0, 082
+ times 16 dw 0, 6483
+ times 16 dw 0, 084
+ times 16 dw 0, 085
+86
+ times 16 dw -1, 487
+ times 16 dw -10, 5888
+ times 16 dw 17, -589
+ times 16 dw 1, 090
+91
+ times 16 dw -1, 492
+ times 16 dw -11, 4093
+ times 16 dw 40, -1194
+ times 16 dw 4, -195
+96
+ times 16 dw 0, 197
+ times 16 dw -5, 1798
+ times 16 dw 58, -1099
+ times 16 dw 4, -1100
+101
+ALIGN 64102
+const tab_LumaCoeffVer_32_avx512, times 32 db 0, 0103
+ times 32 db 0, 64104
+ times 32 db 0, 0105
+ times 32 db 0, 0106
+107
+ times 32 db -1, 4108
+ times 32 db -10, 58109
+ times 32 db 17, -5110
+ times 32 db 1, 0111
+112
+ times 32 db -1, 4113
+ times 32 db -11, 40114
+ times 32 db 40, -11115
+ times 32 db 4, -1116
+117
+ times 32 db 0, 1118
+ times 32 db -5, 17119
+ times 32 db 58, -10120
+ times 32 db 4, -1121
+122
const tab_c_64_n64, times 8 db 64, -64123
124
const interp8_hps_shuf, dd 0, 4, 1, 5, 2, 6, 3, 7125
126
-SECTION .text127
+const interp4_horiz_shuf_load1_avx512, times 2 db 0, 1, 2, 3, 1, 2, 3, 4, 2, 3, 4, 5, 3, 4, 5, 6128
+const interp4_horiz_shuf_load2_avx512, times 2 db 8, 9, 10, 11, 9, 10, 11, 12, 10, 11, 12, 13, 11, 12, 13, 14129
+const interp4_horiz_shuf_load3_avx512, times 2 db 4, 5, 6, 7, 5, 6, 7, 8, 6, 7, 8, 9, 7, 8, 9, 10130
+131
+ALIGN 64132
+interp4_vps_store1_avx512: dq 0, 1, 8, 9, 2, 3, 10, 11133
+interp4_vps_store2_avx512: dq 4, 5, 12, 13, 6, 7, 14, 15134
+const interp4_hps_shuf_avx512, dq 0, 4, 1, 5, 2, 6, 3, 7135
+const interp4_hps_store_16xN_avx512, dq 0, 2, 1, 3, 4, 6, 5, 7136
+const interp8_hps_store_avx512, dq 0, 1, 4, 5, 2, 3, 6, 7137
+const interp8_vsp_store_avx512, dq 0, 2, 4, 6, 1, 3, 5, 7138
139
+SECTION .text140
cextern pb_128141
cextern pw_1142
cextern pw_32143
144
P2S_H_32xN_avx2 48145
146
;-----------------------------------------------------------------------------147
+;p2s and p2s_aligned 32xN avx512 code start148
+;-----------------------------------------------------------------------------149
+150
+%macro PROCESS_P2S_32x4_AVX512 0151
+ pmovzxbw m0, [r0]152
+ pmovzxbw m1, [r0 + r1]153
+ pmovzxbw m2, [r0 + r1 * 2]154
+ pmovzxbw m3, [r0 + r5]155
+156
+ psllw m0, 6157
+ psllw m1, 6158
+ psllw m2, 6159
+ psllw m3, 6160
+ psubw m0, m4161
+ psubw m1, m4162
+ psubw m2, m4163
+ psubw m3, m4164
+165
+ movu [r2], m0166
+ movu [r2 + r3], m1167
+ movu [r2 + r3 * 2], m2168
+ movu [r2 + r6], m3169
+%endmacro170
+171
+;-----------------------------------------------------------------------------172
+; void filterPixelToShort(pixel *src, intptr_t srcStride, int16_t *dst, int16_t dstStride)173
+;-----------------------------------------------------------------------------174
+%if ARCH_X86_64175
+INIT_ZMM avx512176
+cglobal filterPixelToShort_32x8, 3, 7, 5177
+ mov r3d, r3m178
+ add r3d, r3d179
+ lea r5, [r1 * 3]180
+ lea r6, [r3 * 3]181
+182
+ ; load constant183
+ vpbroadcastd m4, [pw_2000]184
+185
+ PROCESS_P2S_32x4_AVX512186
+ lea r0, [r0 + r1 * 4]187
+ lea r2, [r2 + r3 * 4]188
+ PROCESS_P2S_32x4_AVX512189
+ RET190
+191
+INIT_ZMM avx512192
+cglobal filterPixelToShort_32x16, 3, 7, 5193
+ mov r3d, r3m194
+ add r3d, r3d195
+ lea r5, [r1 * 3]196
+ lea r6, [r3 * 3]197
+198
+ ; load constant199
+ vpbroadcastd m4, [pw_2000]200
+201
+%rep 3202
+ PROCESS_P2S_32x4_AVX512203
+ lea r0, [r0 + r1 * 4]204
+ lea r2, [r2 + r3 * 4]205
+%endrep206
+ PROCESS_P2S_32x4_AVX512207
+ RET208
+209
+INIT_ZMM avx512210
+cglobal filterPixelToShort_32x24, 3, 7, 5211
+ mov r3d, r3m212
+ add r3d, r3d213
+ lea r5, [r1 * 3]214
+ lea r6, [r3 * 3]215
+216
+ ; load constant217
+ vpbroadcastd m4, [pw_2000]218
+219
+%rep 5220
+ PROCESS_P2S_32x4_AVX512221
+ lea r0, [r0 + r1 * 4]222
+ lea r2, [r2 + r3 * 4]223
+%endrep224
+ PROCESS_P2S_32x4_AVX512225
+ RET226
+227
+INIT_ZMM avx512228
+cglobal filterPixelToShort_32x32, 3, 7, 5229
+ mov r3d, r3m230
+ add r3d, r3d231
+ lea r5, [r1 * 3]232
+ lea r6, [r3 * 3]233
+234
+ ; load constant235
+ vpbroadcastd m4, [pw_2000]236
+237
+%rep 7238
+ PROCESS_P2S_32x4_AVX512239
+ lea r0, [r0 + r1 * 4]240
+ lea r2, [r2 + r3 * 4]241
+%endrep242
+ PROCESS_P2S_32x4_AVX512243
+ RET244
+245
+INIT_ZMM avx512246
+cglobal filterPixelToShort_32x48, 3, 7, 5247
+ mov r3d, r3m248
+ add r3d, r3d249
+ lea r5, [r1 * 3]250
+ lea r6, [r3 * 3]251
+252
+ ; load constant253
+ vpbroadcastd m4, [pw_2000]254
+255
+%rep 11256
+ PROCESS_P2S_32x4_AVX512257
+ lea r0, [r0 + r1 * 4]258
+ lea r2, [r2 + r3 * 4]259
+%endrep260
+ PROCESS_P2S_32x4_AVX512261
+ RET262
+263
+INIT_ZMM avx512264
+cglobal filterPixelToShort_32x64, 3, 7, 5265
+ mov r3d, r3m266
+ add r3d, r3d267
+ lea r5, [r1 * 3]268
+ lea r6, [r3 * 3]269
+270
+ ; load constant271
+ vpbroadcastd m4, [pw_2000]272
+273
+%rep 15274
+ PROCESS_P2S_32x4_AVX512275
+ lea r0, [r0 + r1 * 4]276
+ lea r2, [r2 + r3 * 4]277
+%endrep278
+ PROCESS_P2S_32x4_AVX512279
+ RET280
+%endif281
+282
+%macro PROCESS_P2S_ALIGNED_32x4_AVX512 0283
+ pmovzxbw m0, [r0]284
+ pmovzxbw m1, [r0 + r1]285
+ pmovzxbw m2, [r0 + r1 * 2]286
+ pmovzxbw m3, [r0 + r5]287
+288
+ psllw m0, 6289
+ psllw m1, 6290
+ psllw m2, 6291
+ psllw m3, 6292
+ psubw m0, m4293
+ psubw m1, m4294
+ psubw m2, m4295
+ psubw m3, m4296
+297
+ mova [r2], m0298
+ mova [r2 + r3], m1299
+ mova [r2 + r3 * 2], m2300
+ mova [r2 + r6], m3301
+%endmacro302
+303
+;-----------------------------------------------------------------------------304
+; void filterPixelToShort(pixel *src, intptr_t srcStride, int16_t *dst, int16_t dstStride)305
+;-----------------------------------------------------------------------------306
+%if ARCH_X86_64307
+INIT_ZMM avx512308
+cglobal filterPixelToShort_aligned_32x8, 3, 7, 5309
+ mov r3d, r3m310
+ add r3d, r3d311
+ lea r5, [r1 * 3]312
+ lea r6, [r3 * 3]313
+314
+ ; load constant315
+ vpbroadcastd m4, [pw_2000]316
+317
+ PROCESS_P2S_ALIGNED_32x4_AVX512318
+ lea r0, [r0 + r1 * 4]319
+ lea r2, [r2 + r3 * 4]320
+ PROCESS_P2S_ALIGNED_32x4_AVX512321
+ RET322
+323
+INIT_ZMM avx512324
+cglobal filterPixelToShort_aligned_32x16, 3, 7, 5325
+ mov r3d, r3m326
+ add r3d, r3d327
+ lea r5, [r1 * 3]328
+ lea r6, [r3 * 3]329
+330
+ ; load constant331
+ vpbroadcastd m4, [pw_2000]332
+333
+%rep 3334
+ PROCESS_P2S_ALIGNED_32x4_AVX512335
+ lea r0, [r0 + r1 * 4]336
+ lea r2, [r2 + r3 * 4]337
+%endrep338
+ PROCESS_P2S_ALIGNED_32x4_AVX512339
+ RET340
+341
+INIT_ZMM avx512342
+cglobal filterPixelToShort_aligned_32x24, 3, 7, 5343
+ mov r3d, r3m344
+ add r3d, r3d345
+ lea r5, [r1 * 3]346
+ lea r6, [r3 * 3]347
+348
+ ; load constant349
+ vpbroadcastd m4, [pw_2000]350
+351
+%rep 5352
+ PROCESS_P2S_ALIGNED_32x4_AVX512353
+ lea r0, [r0 + r1 * 4]354
+ lea r2, [r2 + r3 * 4]355
+%endrep356
+ PROCESS_P2S_ALIGNED_32x4_AVX512357
+ RET358
+359
+INIT_ZMM avx512360
+cglobal filterPixelToShort_aligned_32x32, 3, 7, 5361
+ mov r3d, r3m362
+ add r3d, r3d363
+ lea r5, [r1 * 3]364
+ lea r6, [r3 * 3]365
+366
+ ; load constant367
+ vpbroadcastd m4, [pw_2000]368
+369
+%rep 7370
+ PROCESS_P2S_ALIGNED_32x4_AVX512371
+ lea r0, [r0 + r1 * 4]372
+ lea r2, [r2 + r3 * 4]373
+%endrep374
+ PROCESS_P2S_ALIGNED_32x4_AVX512375
+ RET376
+377
+INIT_ZMM avx512378
+cglobal filterPixelToShort_aligned_32x48, 3, 7, 5379
+ mov r3d, r3m380
+ add r3d, r3d381
+ lea r5, [r1 * 3]382
+ lea r6, [r3 * 3]383
+384
+ ; load constant385
+ vpbroadcastd m4, [pw_2000]386
+387
+%rep 11388
+ PROCESS_P2S_ALIGNED_32x4_AVX512389
+ lea r0, [r0 + r1 * 4]390
+ lea r2, [r2 + r3 * 4]391
+%endrep392
+ PROCESS_P2S_ALIGNED_32x4_AVX512393
+ RET394
+395
+INIT_ZMM avx512396
+cglobal filterPixelToShort_aligned_32x64, 3, 7, 5397
+ mov r3d, r3m398
+ add r3d, r3d399
+ lea r5, [r1 * 3]400
+ lea r6, [r3 * 3]401
+402
+ ; load constant403
+ vpbroadcastd m4, [pw_2000]404
+405
+%rep 15406
+ PROCESS_P2S_ALIGNED_32x4_AVX512407
+ lea r0, [r0 + r1 * 4]408
+ lea r2, [r2 + r3 * 4]409
+%endrep410
+ PROCESS_P2S_ALIGNED_32x4_AVX512411
+ RET412
+%endif413
+;-----------------------------------------------------------------------------414
+;p2s and p2s_aligned 32xN avx512 code end415
+;-----------------------------------------------------------------------------416
+;-----------------------------------------------------------------------------417
; void filterPixelToShort(pixel *src, intptr_t srcStride, int16_t *dst, int16_t dstStride)418
;-----------------------------------------------------------------------------419
%macro P2S_H_64xN 1420
421
P2S_H_64xN_avx2 48422
423
;-----------------------------------------------------------------------------424
+;p2s and p2s_aligned 64xN avx512 code start425
+;-----------------------------------------------------------------------------426
+%macro PROCESS_P2S_64x4_AVX512 0427
+ pmovzxbw m0, [r0]428
+ pmovzxbw m1, [r0 + mmsize/2]429
+ pmovzxbw m2, [r0 + r1]430
+ pmovzxbw m3, [r0 + r1 + mmsize/2]431
+432
+ psllw m0, 6433
+ psllw m1, 6434
+ psllw m2, 6435
+ psllw m3, 6436
+ psubw m0, m4437
+ psubw m1, m4438
+ psubw m2, m4439
+ psubw m3, m4440
+ movu [r2], m0441
+ movu [r2 + mmsize], m1442
+ movu [r2 + r3], m2443
+ movu [r2 + r3 + mmsize], m3444
+445
+ pmovzxbw m0, [r0 + r1 * 2]446
+ pmovzxbw m1, [r0 + r1 * 2 + mmsize/2]447
+ pmovzxbw m2, [r0 + r5]448
+ pmovzxbw m3, [r0 + r5 + mmsize/2]449
+450
+ psllw m0, 6451
+ psllw m1, 6452
+ psllw m2, 6453
+ psllw m3, 6454
+ psubw m0, m4455
+ psubw m1, m4456
+ psubw m2, m4457
+ psubw m3, m4458
+ movu [r2 + r3 * 2], m0459
+ movu [r2 + r3 * 2 + mmsize], m1460
+ movu [r2 + r6], m2461
+ movu [r2 + r6 + mmsize], m3462
+%endmacro463
+464
+%macro PROCESS_P2S_ALIGNED_64x4_AVX512 0465
+ pmovzxbw m0, [r0]466
+ pmovzxbw m1, [r0 + mmsize/2]467
+ pmovzxbw m2, [r0 + r1]468
+ pmovzxbw m3, [r0 + r1 + mmsize/2]469
+470
+ psllw m0, 6471
+ psllw m1, 6472
+ psllw m2, 6473
+ psllw m3, 6474
+ psubw m0, m4475
+ psubw m1, m4476
+ psubw m2, m4477
+ psubw m3, m4478
+ mova [r2], m0479
+ mova [r2 + mmsize], m1480
+ mova [r2 + r3], m2481
+ mova [r2 + r3 + mmsize], m3482
+483
+ pmovzxbw m0, [r0 + r1 * 2]484
+ pmovzxbw m1, [r0 + r1 * 2 + mmsize/2]485
+ pmovzxbw m2, [r0 + r5]486
+ pmovzxbw m3, [r0 + r5 + mmsize/2]487
+488
+ psllw m0, 6489
+ psllw m1, 6490
+ psllw m2, 6491
+ psllw m3, 6492
+ psubw m0, m4493
+ psubw m1, m4494
+ psubw m2, m4495
+ psubw m3, m4496
+ mova [r2 + r3 * 2], m0497
+ mova [r2 + r3 * 2 + mmsize], m1498
+ mova [r2 + r6], m2499
+ mova [r2 + r6 + mmsize], m3500
+%endmacro501
+;-----------------------------------------------------------------------------502
+; void filterPixelToShort(pixel *src, intptr_t srcStride, int16_t *dst, int16_t dstStride)503
+;-----------------------------------------------------------------------------504
+%if ARCH_X86_64505
+INIT_ZMM avx512506
+cglobal filterPixelToShort_64x64, 3, 7, 5507
+ mov r3d, r3m508
+ add r3d, r3d509
+ lea r5, [r1 * 3]510
+ lea r6, [r3 * 3]511
+512
+ ; load constant513
+ vpbroadcastd m4, [pw_2000]514
+515
+%rep 15516
+ PROCESS_P2S_64x4_AVX512517
+ lea r0, [r0 + r1 * 4]518
+ lea r2, [r2 + r3 * 4]519
+%endrep520
+ PROCESS_P2S_64x4_AVX512521
+ RET522
+523
+INIT_ZMM avx512524
+cglobal filterPixelToShort_64x48, 3, 7, 5525
+ mov r3d, r3m526
+ add r3d, r3d527
+ lea r5, [r1 * 3]528
+ lea r6, [r3 * 3]529
+530
+ ; load constant531
+ vpbroadcastd m4, [pw_2000]532
+533
+%rep 11534
+ PROCESS_P2S_64x4_AVX512535
+ lea r0, [r0 + r1 * 4]536
+ lea r2, [r2 + r3 * 4]537
+%endrep538
+ PROCESS_P2S_64x4_AVX512539
+ RET540
+541
+INIT_ZMM avx512542
+cglobal filterPixelToShort_64x32, 3, 7, 5543
+ mov r3d, r3m544
+ add r3d, r3d545
+ lea r5, [r1 * 3]546
+ lea r6, [r3 * 3]547
+548
+ ; load constant549
+ vpbroadcastd m4, [pw_2000]550
+551
+%rep 7552
+ PROCESS_P2S_64x4_AVX512553
+ lea r0, [r0 + r1 * 4]554
+ lea r2, [r2 + r3 * 4]555
+%endrep556
+ PROCESS_P2S_64x4_AVX512557
+ RET558
+559
+INIT_ZMM avx512560
+cglobal filterPixelToShort_64x16, 3, 7, 5561
+ mov r3d, r3m562
+ add r3d, r3d563
+ lea r5, [r1 * 3]564
+ lea r6, [r3 * 3]565
+566
+ ; load constant567
+ vpbroadcastd m4, [pw_2000]568
+569
+%rep 3570
+ PROCESS_P2S_64x4_AVX512571
+ lea r0, [r0 + r1 * 4]572
+ lea r2, [r2 + r3 * 4]573
+%endrep574
+ PROCESS_P2S_64x4_AVX512575
+ RET576
+577
+INIT_ZMM avx512578
+cglobal filterPixelToShort_aligned_64x64, 3, 7, 5579
+ mov r3d, r3m580
+ add r3d, r3d581
+ lea r5, [r1 * 3]582
+ lea r6, [r3 * 3]583
+584
+ ; load constant585
+ vpbroadcastd m4, [pw_2000]586
+587
+%rep 15588
+ PROCESS_P2S_ALIGNED_64x4_AVX512589
+ lea r0, [r0 + r1 * 4]590
+ lea r2, [r2 + r3 * 4]591
+%endrep592
+ PROCESS_P2S_ALIGNED_64x4_AVX512593
+ RET594
+595
+INIT_ZMM avx512596
+cglobal filterPixelToShort_aligned_64x48, 3, 7, 5597
+ mov r3d, r3m598
+ add r3d, r3d599
+ lea r5, [r1 * 3]600
+ lea r6, [r3 * 3]601
+602
+ ; load constant603
+ vpbroadcastd m4, [pw_2000]604
+605
+%rep 11606
+ PROCESS_P2S_ALIGNED_64x4_AVX512607
+ lea r0, [r0 + r1 * 4]608
+ lea r2, [r2 + r3 * 4]609
+%endrep610
+ PROCESS_P2S_ALIGNED_64x4_AVX512611
+ RET612
+613
+INIT_ZMM avx512614
+cglobal filterPixelToShort_aligned_64x32, 3, 7, 5615
+ mov r3d, r3m616
+ add r3d, r3d617
+ lea r5, [r1 * 3]618
+ lea r6, [r3 * 3]619
+620
+ ; load constant621
+ vpbroadcastd m4, [pw_2000]622
+623
+%rep 7624
+ PROCESS_P2S_ALIGNED_64x4_AVX512625
+ lea r0, [r0 + r1 * 4]626
+ lea r2, [r2 + r3 * 4]627
+%endrep628
+ PROCESS_P2S_ALIGNED_64x4_AVX512629
+ RET630
+631
+INIT_ZMM avx512632
+cglobal filterPixelToShort_aligned_64x16, 3, 7, 5633
+ mov r3d, r3m634
+ add r3d, r3d635
+ lea r5, [r1 * 3]636
+ lea r6, [r3 * 3]637
+638
+ ; load constant639
+ vpbroadcastd m4, [pw_2000]640
+641
+%rep 3642
+ PROCESS_P2S_ALIGNED_64x4_AVX512643
+ lea r0, [r0 + r1 * 4]644
+ lea r2, [r2 + r3 * 4]645
+%endrep646
+ PROCESS_P2S_ALIGNED_64x4_AVX512647
+ RET648
+%endif649
+;-----------------------------------------------------------------------------650
+;p2s and p2s_aligned 64xN avx512 code end651
+;-----------------------------------------------------------------------------652
+653
+;-----------------------------------------------------------------------------654
; void filterPixelToShort(pixel src, intptr_t srcStride, int16_t dst, int16_t dstStride)655
;-----------------------------------------------------------------------------656
%macro P2S_H_12xN 1657
658
jnz .loop659
RET660
661
+;-----------------------------------------------------------------------------662
+;p2s and p2s_aligned 48xN avx512 code start663
+;-----------------------------------------------------------------------------664
+%macro PROCESS_P2S_48x8_AVX512 0665
+ pmovzxbw m0, [r0]666
+ pmovzxbw m1, [r0 + r1]667
+ pmovzxbw m2, [r0 + r1 * 2]668
+ pmovzxbw m3, [r0 + r5]669
+ psllw m0, 6670
+ psllw m1, 6671
+ psllw m2, 6672
+ psllw m3, 6673
+ psubw m0, m4674
+ psubw m1, m4675
+ psubw m2, m4676
+ psubw m3, m4677
+ movu [r2], m0678
+ movu [r2 + r3], m1679
+ movu [r2 + r3 * 2], m2680
+ movu [r2 + r6], m3681
+682
+ pmovzxbw ym0, [r0 + 32]683
+ pmovzxbw ym1, [r0 + r1 + 32]684
+ pmovzxbw ym2, [r0 + r1 * 2 + 32]685
+ pmovzxbw ym3, [r0 + r5 + 32]686
+ psllw ym0, 6687
+ psllw ym1, 6688
+ psllw ym2, 6689
+ psllw ym3, 6690
+ psubw ym0, ym4691
+ psubw ym1, ym4692
+ psubw ym2, ym4693
+ psubw ym3, ym4694
+ movu [r2 + 64], ym0695
+ movu [r2 + r3 + 64], ym1696
+ movu [r2 + r3 * 2 + 64], ym2697
+ movu [r2 + r6 + 64], ym3698
+699
+ lea r0, [r0 + r1 * 4]700
+ lea r2, [r2 + r3 * 4]701
+702
+ pmovzxbw m0, [r0]703
+ pmovzxbw m1, [r0 + r1]704
+ pmovzxbw m2, [r0 + r1 * 2]705
+ pmovzxbw m3, [r0 + r5]706
+ psllw m0, 6707
+ psllw m1, 6708
+ psllw m2, 6709
+ psllw m3, 6710
+ psubw m0, m4711
+ psubw m1, m4712
+ psubw m2, m4713
+ psubw m3, m4714
+ movu [r2], m0715
+ movu [r2 + r3], m1716
+ movu [r2 + r3 * 2], m2717
+ movu [r2 + r6], m3718
+719
+ pmovzxbw ym0, [r0 + 32]720
+ pmovzxbw ym1, [r0 + r1 + 32]721
+ pmovzxbw ym2, [r0 + r1 * 2 + 32]722
+ pmovzxbw ym3, [r0 + r5 + 32]723
+ psllw ym0, 6724
+ psllw ym1, 6725
+ psllw ym2, 6726
+ psllw ym3, 6727
+ psubw ym0, ym4728
+ psubw ym1, ym4729
+ psubw ym2, ym4730
+ psubw ym3, ym4731
+ movu [r2 + 64], ym0732
+ movu [r2 + r3 + 64], ym1733
+ movu [r2 + r3 * 2 + 64], ym2734
+ movu [r2 + r6 + 64], ym3735
+%endmacro736
+737
+%macro PROCESS_P2S_ALIGNED_48x8_AVX512 0738
+ pmovzxbw m0, [r0]739
+ pmovzxbw m1, [r0 + r1]740
+ pmovzxbw m2, [r0 + r1 * 2]741
+ pmovzxbw m3, [r0 + r5]742
+ psllw m0, 6743
+ psllw m1, 6744
+ psllw m2, 6745
+ psllw m3, 6746
+ psubw m0, m4747
+ psubw m1, m4748
+ psubw m2, m4749
+ psubw m3, m4750
+ mova [r2], m0751
+ mova [r2 + r3], m1752
+ mova [r2 + r3 * 2], m2753
+ mova [r2 + r6], m3754
+755
+ pmovzxbw ym0, [r0 + 32]756
+ pmovzxbw ym1, [r0 + r1 + 32]757
+ pmovzxbw ym2, [r0 + r1 * 2 + 32]758
+ pmovzxbw ym3, [r0 + r5 + 32]759
+ psllw ym0, 6760
+ psllw ym1, 6761
+ psllw ym2, 6762
+ psllw ym3, 6763
+ psubw ym0, ym4764
+ psubw ym1, ym4765
+ psubw ym2, ym4766
+ psubw ym3, ym4767
+ mova [r2 + 64], ym0768
+ mova [r2 + r3 + 64], ym1769
+ mova [r2 + r3 * 2 + 64], ym2770
+ mova [r2 + r6 + 64], ym3771
+772
+ lea r0, [r0 + r1 * 4]773
+ lea r2, [r2 + r3 * 4]774
+775
+ pmovzxbw m0, [r0]776
+ pmovzxbw m1, [r0 + r1]777
+ pmovzxbw m2, [r0 + r1 * 2]778
+ pmovzxbw m3, [r0 + r5]779
+ psllw m0, 6780
+ psllw m1, 6781
+ psllw m2, 6782
+ psllw m3, 6783
+ psubw m0, m4784
+ psubw m1, m4785
+ psubw m2, m4786
+ psubw m3, m4787
+ mova [r2], m0788
+ mova [r2 + r3], m1789
+ mova [r2 + r3 * 2], m2790
+ mova [r2 + r6], m3791
+792
+ pmovzxbw ym0, [r0 + 32]793
+ pmovzxbw ym1, [r0 + r1 + 32]794
+ pmovzxbw ym2, [r0 + r1 * 2 + 32]795
+ pmovzxbw ym3, [r0 + r5 + 32]796
+ psllw ym0, 6797
+ psllw ym1, 6798
+ psllw ym2, 6799
+ psllw ym3, 6800
+ psubw ym0, ym4801
+ psubw ym1, ym4802
+ psubw ym2, ym4803
+ psubw ym3, ym4804
+ mova [r2 + 64], ym0805
+ mova [r2 + r3 + 64], ym1806
+ mova [r2 + r3 * 2 + 64], ym2807
+ mova [r2 + r6 + 64], ym3808
+%endmacro809
+;-----------------------------------------------------------------------------810
+; void filterPixelToShort(pixel *src, intptr_t srcStride, int16_t *dst, int16_t dstStride)811
+;-----------------------------------------------------------------------------812
+%if ARCH_X86_64813
+INIT_ZMM avx512814
+cglobal filterPixelToShort_48x64, 3,7,5815
+ mov r3d, r3m816
+ add r3d, r3d817
+ lea r5, [r1 * 3]818
+ lea r6, [r3 * 3]819
+820
+ ; load constant821
+ vpbroadcastd m4, [pw_2000]822
+823
+ PROCESS_P2S_48x8_AVX512824
+ lea r0, [r0 + r1 * 4]825
+ lea r2, [r2 + r3 * 4]826
+ PROCESS_P2S_48x8_AVX512827
+ lea r0, [r0 + r1 * 4]828
+ lea r2, [r2 + r3 * 4]829
+ PROCESS_P2S_48x8_AVX512830
+ lea r0, [r0 + r1 * 4]831
+ lea r2, [r2 + r3 * 4]832
+ PROCESS_P2S_48x8_AVX512833
+ lea r0, [r0 + r1 * 4]834
+ lea r2, [r2 + r3 * 4]835
+ PROCESS_P2S_48x8_AVX512836
+ lea r0, [r0 + r1 * 4]837
+ lea r2, [r2 + r3 * 4]838
+ PROCESS_P2S_48x8_AVX512839
+ lea r0, [r0 + r1 * 4]840
+ lea r2, [r2 + r3 * 4]841
+ PROCESS_P2S_48x8_AVX512842
+ lea r0, [r0 + r1 * 4]843
+ lea r2, [r2 + r3 * 4]844
+ PROCESS_P2S_48x8_AVX512845
+ RET846
+847
+INIT_ZMM avx512848
+cglobal filterPixelToShort_aligned_48x64, 3,7,5849
+ mov r3d, r3m850
+ add r3d, r3d851
+ lea r5, [r1 * 3]852
+ lea r6, [r3 * 3]853
+854
+ ; load constant855
+ vpbroadcastd m4, [pw_2000]856
+857
+ PROCESS_P2S_ALIGNED_48x8_AVX512858
+ lea r0, [r0 + r1 * 4]859
+ lea r2, [r2 + r3 * 4]860
+ PROCESS_P2S_ALIGNED_48x8_AVX512861
+ lea r0, [r0 + r1 * 4]862
+ lea r2, [r2 + r3 * 4]863
+ PROCESS_P2S_ALIGNED_48x8_AVX512864
+ lea r0, [r0 + r1 * 4]865
+ lea r2, [r2 + r3 * 4]866
+ PROCESS_P2S_ALIGNED_48x8_AVX512867
+ lea r0, [r0 + r1 * 4]868
+ lea r2, [r2 + r3 * 4]869
+ PROCESS_P2S_ALIGNED_48x8_AVX512870
+ lea r0, [r0 + r1 * 4]871
+ lea r2, [r2 + r3 * 4]872
+ PROCESS_P2S_ALIGNED_48x8_AVX512873
+ lea r0, [r0 + r1 * 4]874
+ lea r2, [r2 + r3 * 4]875
+ PROCESS_P2S_ALIGNED_48x8_AVX512876
+ lea r0, [r0 + r1 * 4]877
+ lea r2, [r2 + r3 * 4]878
+ PROCESS_P2S_ALIGNED_48x8_AVX512879
+ RET880
+%endif881
+;-----------------------------------------------------------------------------882
+;p2s and p2s_aligned 48xN avx512 code end883
+;-----------------------------------------------------------------------------884
885
%macro PROCESS_LUMA_W4_4R 0886
movd m0, [r0]887
888
889
FILTER_VER_LUMA_S_AVX2_32x24 sp890
FILTER_VER_LUMA_S_AVX2_32x24 ss891
+;-------------------------------------------------------------------------------------------------------------892
+;ipfilter_chroma_avx512 code start893
+;-------------------------------------------------------------------------------------------------------------894
+%macro PROCESS_IPFILTER_CHROMA_PP_64x1_AVX512 0895
+ ; register map896
+ ; m0 - interpolate coeff897
+ ; m1, m2 - shuffle order table898
+ ; m3 - constant word 1899
+ ; m4 - constant word 512900
+901
+ movu m5, [r0]902
+ pshufb m6, m5, m2903
+ pshufb m5, m5, m1904
+ pmaddubsw m5, m0905
+ pmaddubsw m6, m0906
+ pmaddwd m5, m3907
+ pmaddwd m6, m3908
+909
+ movu m7, [r0 + 4]910
+ pshufb m8, m7, m2911
+ pshufb m7, m7, m1912
+ pmaddubsw m7, m0913
+ pmaddubsw m8, m0914
+ pmaddwd m7, m3915
+ pmaddwd m8, m3916
+917
+ packssdw m5, m7918
+ packssdw m6, m8919
+ pmulhrsw m5, m4920
+ pmulhrsw m6, m4921
+ packuswb m5, m6922
+ movu [r2], m5923
+%endmacro924
+925
+%macro PROCESS_IPFILTER_CHROMA_PP_32x2_AVX512 0926
+ ; register map927
+ ; m0 - interpolate coeff928
+ ; m1, m2 - shuffle order table929
+ ; m3 - constant word 1930
+ ; m4 - constant word 512931
+ ; m9 - store shuffle order table932
+933
+ movu ym5, [r0]934
+ vinserti32x8 m5, [r0 + r1], 1935
+ movu ym7, [r0 + 4]936
+ vinserti32x8 m7, [r0 + r1 + 4], 1937
+938
+ pshufb m6, m5, m2939
+ pshufb m5, m1940
+ pshufb m8, m7, m2941
+ pshufb m7, m1942
+943
+ pmaddubsw m5, m0944
+ pmaddubsw m7, m0945
+ pmaddwd m5, m3946
+ pmaddwd m7, m3947
+948
+ pmaddubsw m6, m0949
+ pmaddubsw m8, m0950
+ pmaddwd m6, m3951
+ pmaddwd m8, m3952
+953
+ packssdw m5, m7954
+ packssdw m6, m8955
+ pmulhrsw m5, m4956
+ pmulhrsw m6, m4957
+ packuswb m5, m6958
+ movu [r2], ym5959
+ vextracti32x8 [r2 + r3], m5, 1960
+%endmacro961
+962
+%macro PROCESS_IPFILTER_CHROMA_PP_16x4_AVX512 0963
+ ; register map964
+ ; m0 - interpolate coeff965
+ ; m1, m2 - shuffle order table966
+ ; m3 - constant word 1967
+ ; m4 - constant word 512968
+969
+ movu xm5, [r0]970
+ vinserti32x4 m5, [r0 + r1], 1971
+ vinserti32x4 m5, [r0 + 2 * r1], 2972
+ vinserti32x4 m5, [r0 + r6], 3973
+ pshufb m6, m5, m2974
+ pshufb m5, m1975
+976
+ movu xm7, [r0 + 4]977
+ vinserti32x4 m7, [r0 + r1 + 4], 1978
+ vinserti32x4 m7, [r0 + 2 * r1 + 4], 2979
+ vinserti32x4 m7, [r0 + r6 + 4], 3980
+ pshufb m8, m7, m2981
+ pshufb m7, m1982
+983
+ pmaddubsw m5, m0984
+ pmaddubsw m7, m0985
+ pmaddwd m5, m3986
+ pmaddwd m7, m3987
+988
+ pmaddubsw m6, m0989
+ pmaddubsw m8, m0990
+ pmaddwd m6, m3991
+ pmaddwd m8, m3992
+993
+ packssdw m5, m7994
+ packssdw m6, m8995
+ pmulhrsw m5, m4996
+ pmulhrsw m6, m4997
+ packuswb m5, m6998
+ movu [r2], xm5999
+ vextracti32x4 [r2 + r3], m5, 11000
+ vextracti32x4 [r2 + 2 * r3], m5, 21001
+ vextracti32x4 [r2 + r7], m5, 31002
+%endmacro1003
+1004
+%macro PROCESS_IPFILTER_CHROMA_PP_48x4_AVX512 01005
+ ; register map1006
+ ; m0 - interpolate coeff1007
+ ; m1, m2 - shuffle order table1008
+ ; m3 - constant word 11009
+ ; m4 - constant word 5121010
+ movu ym5, [r0]1011
+ vinserti32x8 m5, [r0 + r1], 11012
+ movu ym7, [r0 + 4]1013
+ vinserti32x8 m7, [r0 + r1 + 4], 11014
+1015
+ pshufb m6, m5, m21016
+ pshufb m5, m11017
+ pshufb m8, m7, m21018
+ pshufb m7, m11019
+1020
+ pmaddubsw m5, m01021
+ pmaddubsw m7, m01022
+ pmaddwd m5, m31023
+ pmaddwd m7, m31024
+1025
+ pmaddubsw m6, m01026
+ pmaddubsw m8, m01027
+ pmaddwd m6, m31028
+ pmaddwd m8, m31029
+1030
+ packssdw m5, m71031
+ packssdw m6, m81032
+ pmulhrsw m5, m41033
+ pmulhrsw m6, m41034
+ packuswb m5, m61035
+ movu [r2], ym51036
+ vextracti32x8 [r2 + r3], m5, 11037
+1038
+ movu ym5, [r0 + 2 * r1]1039
+ vinserti32x8 m5, [r0 + r6], 11040
+ movu ym7, [r0 + 2 * r1 + 4]1041
+ vinserti32x8 m7, [r0 + r6 + 4], 11042
+1043
+ pshufb m6, m5, m21044
+ pshufb m5, m11045
+ pshufb m8, m7, m21046
+ pshufb m7, m11047
+1048
+ pmaddubsw m5, m01049
+ pmaddubsw m7, m01050
+ pmaddwd m5, m31051
+ pmaddwd m7, m31052
+1053
+ pmaddubsw m6, m01054
+ pmaddubsw m8, m01055
+ pmaddwd m6, m31056
+ pmaddwd m8, m31057
+1058
+ packssdw m5, m71059
+ packssdw m6, m81060
+ pmulhrsw m5, m41061
+ pmulhrsw m6, m41062
+ packuswb m5, m61063
+ movu [r2 + 2 * r3], ym51064
+ vextracti32x8 [r2 + r7], m5, 11065
+1066
+ movu xm5, [r0 + mmsize/2]1067
+ vinserti32x4 m5, [r0 + r1 + mmsize/2], 11068
+ vinserti32x4 m5, [r0 + 2 * r1 + mmsize/2], 21069
+ vinserti32x4 m5, [r0 + r6 + mmsize/2], 31070
+ pshufb m6, m5, m21071
+ pshufb m5, m11072
+1073
+ movu xm7, [r0 + 36]1074
+ vinserti32x4 m7, [r0 + r1 + 36], 11075
+ vinserti32x4 m7, [r0 + 2 * r1 + 36], 21076
+ vinserti32x4 m7, [r0 + r6 + 36], 31077
+ pshufb m8, m7, m21078
+ pshufb m7, m11079
+1080
+ pmaddubsw m5, m01081
+ pmaddubsw m7, m01082
+ pmaddwd m5, m31083
+ pmaddwd m7, m31084
+1085
+ pmaddubsw m6, m01086
+ pmaddubsw m8, m01087
+ pmaddwd m6, m31088
+ pmaddwd m8, m31089
+1090
+ packssdw m5, m71091
+ packssdw m6, m81092
+ pmulhrsw m5, m41093
+ pmulhrsw m6, m41094
+ packuswb m5, m61095
+ movu [r2 + mmsize/2], xm51096
+ vextracti32x4 [r2 + r3 + mmsize/2], m5, 11097
+ vextracti32x4 [r2 + 2 * r3 + mmsize/2], m5, 21098
+ vextracti32x4 [r2 + r7 + mmsize/2], m5, 31099
+%endmacro1100
+1101
+;-------------------------------------------------------------------------------------------------------------1102
+; void interp_4tap_horiz_pp_64xN(pixel *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx1103
+;-------------------------------------------------------------------------------------------------------------1104
+%macro IPFILTER_CHROMA_PP_64xN_AVX512 11105
+INIT_ZMM avx5121106
+cglobal interp_4tap_horiz_pp_64x%1, 4,6,91107
+ mov r4d, r4m1108
+1109
+%ifdef PIC1110
+ lea r5, [tab_ChromaCoeff]1111
+ vpbroadcastd m0, [r5 + r4 * 4]1112
+%else1113
+ vpbroadcastd m0, [tab_ChromaCoeff + r4 * 4]1114
+%endif1115
+1116
+ vbroadcasti32x8 m1, [interp4_horiz_shuf_load1_avx512]1117
+ vbroadcasti32x8 m2, [interp4_horiz_shuf_load2_avx512]1118
+ vbroadcasti32x8 m3, [pw_1]1119
+ vbroadcasti32x8 m4, [pw_512]1120
+ dec r01121
+1122
+%rep %1 - 11123
+ PROCESS_IPFILTER_CHROMA_PP_64x1_AVX5121124
+ lea r2, [r2 + r3]1125
+ lea r0, [r0 + r1]1126
+%endrep1127
+ PROCESS_IPFILTER_CHROMA_PP_64x1_AVX5121128
+ RET1129
+%endmacro1130
+1131
+%if ARCH_X86_641132
+ IPFILTER_CHROMA_PP_64xN_AVX512 641133
+ IPFILTER_CHROMA_PP_64xN_AVX512 321134
+ IPFILTER_CHROMA_PP_64xN_AVX512 481135
+ IPFILTER_CHROMA_PP_64xN_AVX512 161136
+%endif1137
+1138
+%macro IPFILTER_CHROMA_PP_32xN_AVX512 11139
+INIT_ZMM avx5121140
+cglobal interp_4tap_horiz_pp_32x%1, 4,6,91141
+ mov r4d, r4m1142
+1143
+%ifdef PIC1144
+ lea r5, [tab_ChromaCoeff]1145
+ vpbroadcastd m0, [r5 + r4 * 4]1146
+%else1147
+ vpbroadcastd m0, [tab_ChromaCoeff + r4 * 4]1148
+%endif1149
+1150
+ vbroadcasti32x8 m1, [interp4_horiz_shuf_load1_avx512]1151
+ vbroadcasti32x8 m2, [interp4_horiz_shuf_load2_avx512]1152
+ vbroadcasti32x8 m3, [pw_1]1153
+ vbroadcasti32x8 m4, [pw_512]1154
+ dec r01155
+1156
+%rep %1/2 - 11157
+ PROCESS_IPFILTER_CHROMA_PP_32x2_AVX5121158
+ lea r2, [r2 + 2 * r3]1159
+ lea r0, [r0 + 2 * r1]1160
+%endrep1161
+ PROCESS_IPFILTER_CHROMA_PP_32x2_AVX5121162
+ RET1163
+%endmacro1164
+1165
+%if ARCH_X86_641166
+ IPFILTER_CHROMA_PP_32xN_AVX512 161167
+ IPFILTER_CHROMA_PP_32xN_AVX512 241168
+ IPFILTER_CHROMA_PP_32xN_AVX512 81169
+ IPFILTER_CHROMA_PP_32xN_AVX512 321170
+ IPFILTER_CHROMA_PP_32xN_AVX512 641171
+ IPFILTER_CHROMA_PP_32xN_AVX512 481172
+%endif1173
+1174
+%macro IPFILTER_CHROMA_PP_16xN_AVX512 11175
+INIT_ZMM avx5121176
+cglobal interp_4tap_horiz_pp_16x%1, 4,8,91177
+ mov r4d, r4m1178
+ lea r6, [3 * r1]1179
+ lea r7, [3 * r3]1180
+%ifdef PIC1181
+ lea r5, [tab_ChromaCoeff]1182
+ vpbroadcastd m0, [r5 + r4 * 4]1183
+%else1184
+ vpbroadcastd m0, [tab_ChromaCoeff + r4 * 4]1185
+%endif1186
+1187
+ vbroadcasti32x8 m1, [interp4_horiz_shuf_load1_avx512]1188
+ vbroadcasti32x8 m2, [interp4_horiz_shuf_load2_avx512]1189
+ vbroadcasti32x8 m3, [pw_1]1190
+ vbroadcasti32x8 m4, [pw_512]1191
+ dec r01192
+1193
+%rep %1/4 - 11194
+ PROCESS_IPFILTER_CHROMA_PP_16x4_AVX5121195
+ lea r2, [r2 + 4 * r3]1196
+ lea r0, [r0 + 4 * r1]1197
+%endrep1198
+ PROCESS_IPFILTER_CHROMA_PP_16x4_AVX5121199
+ RET1200
+%endmacro1201
+1202
+%if ARCH_X86_641203
+ IPFILTER_CHROMA_PP_16xN_AVX512 41204
+ IPFILTER_CHROMA_PP_16xN_AVX512 81205
+ IPFILTER_CHROMA_PP_16xN_AVX512 121206
+ IPFILTER_CHROMA_PP_16xN_AVX512 161207
+ IPFILTER_CHROMA_PP_16xN_AVX512 241208
+ IPFILTER_CHROMA_PP_16xN_AVX512 321209
+ IPFILTER_CHROMA_PP_16xN_AVX512 641210
+%endif1211
+1212
+%if ARCH_X86_641213
+INIT_ZMM avx5121214
+cglobal interp_4tap_horiz_pp_48x64, 4,8,91215
+ mov r4d, r4m1216
+ lea r6, [3 * r1]1217
+ lea r7, [3 * r3]1218
+%ifdef PIC1219
+ lea r5, [tab_ChromaCoeff]1220
+ vpbroadcastd m0, [r5 + r4 * 4]1221
+%else1222
+ vpbroadcastd m0, [tab_ChromaCoeff + r4 * 4]1223
+%endif1224
+1225
+ vbroadcasti32x8 m1, [interp4_horiz_shuf_load1_avx512]1226
+ vbroadcasti32x8 m2, [interp4_horiz_shuf_load2_avx512]1227
+ vbroadcasti32x8 m3, [pw_1]1228
+ vbroadcasti32x8 m4, [pw_512]1229
+ dec r01230
+1231
+%rep 151232
+ PROCESS_IPFILTER_CHROMA_PP_48x4_AVX5121233
+ lea r2, [r2 + 4 * r3]1234
+ lea r0, [r0 + 4 * r1]1235
+%endrep1236
+ PROCESS_IPFILTER_CHROMA_PP_48x4_AVX5121237
+ RET1238
+%endif1239
+1240
+%macro PROCESS_IPFILTER_CHROMA_PS_64x1_AVX512 01241
+ movu ym6, [r0]1242
+ vinserti32x8 m6, [r0 + 4], 11243
+ pshufb m7, m6, m21244
+ pshufb m6, m11245
+ pmaddubsw m6, m01246
+ pmaddubsw m7, m01247
+ pmaddwd m6, m31248
+ pmaddwd m7, m31249
+1250
+ movu ym8, [r0 + 32]1251
+ vinserti32x8 m8, [r0 + 36], 11252
+ pshufb m9, m8, m21253
+ pshufb m8, m11254
+ pmaddubsw m8, m01255
+ pmaddubsw m9, m01256
+ pmaddwd m8, m31257
+ pmaddwd m9, m31258
+1259
+ packssdw m6, m71260
+ packssdw m8, m91261
+ psubw m6, m41262
+ psubw m8, m41263
+ vpermq m6, m10, m61264
+ vpermq m8, m10, m81265
+ movu [r2], m61266
+ movu [r2 + mmsize],m81267
+%endmacro1268
+1269
+;-------------------------------------------------------------------------------------------------------------1270
+; void interp_horiz_ps_64xN(const pixel* src, intptr_t srcStride, int16_t* dst, intptr_t dstStride, int coeffIdx, int isRowExt)1271
+;-------------------------------------------------------------------------------------------------------------1272
+%macro IPFILTER_CHROMA_PS_64xN_AVX512 11273
+INIT_ZMM avx5121274
+cglobal interp_4tap_horiz_ps_64x%1, 4,7,111275
+ mov r4d, r4m1276
+ mov r5d, r5m1277
+1278
+%ifdef PIC1279
+ lea r6, [tab_ChromaCoeff]1280
+ vpbroadcastd m0, [r6 + r4 * 4]1281
+%else1282
+ vpbroadcastd m0, [tab_ChromaCoeff + r4 * 4]1283
+%endif1284
+1285
+ vbroadcasti32x8 m1, [interp4_horiz_shuf_load1_avx512]1286
+ vbroadcasti32x8 m2, [interp4_horiz_shuf_load2_avx512]1287
+ vbroadcasti32x8 m3, [pw_1]1288
+ vbroadcasti32x8 m4, [pw_2000]1289
+ mova m10, [interp4_hps_shuf_avx512]1290
+1291
+ ; register map1292
+ ; m0 - interpolate coeff1293
+ ; m1,m2 - load shuffle order table1294
+ ; m3 - constant word 11295
+ ; m4 - constant word 20001296
+ ; m10 - store shuffle order table1297
+1298
+ mov r6d, %11299
+ dec r01300
+ test r5d, r5d1301
+ je .loop1302
+ sub r0, r11303
+ add r6d, 31304
+1305
+.loop:1306
+ PROCESS_IPFILTER_CHROMA_PS_64x1_AVX5121307
+ lea r2, [r2 + 2 * r3]1308
+ lea r0, [r0 + r1]1309
+ dec r6d1310
+ jnz .loop1311
+ RET1312
+%endmacro1313
+1314
+%if ARCH_X86_641315
+ IPFILTER_CHROMA_PS_64xN_AVX512 641316
+ IPFILTER_CHROMA_PS_64xN_AVX512 321317
+ IPFILTER_CHROMA_PS_64xN_AVX512 481318
+ IPFILTER_CHROMA_PS_64xN_AVX512 161319
+%endif1320
+1321
+%macro PROCESS_IPFILTER_CHROMA_PS_32x1_AVX512 01322
+ movu ym6, [r0]1323
+ vinserti32x8 m6, [r0 + 4], 11324
+ pshufb m7, m6, m21325
+ pshufb m6, m6, m11326
+ pmaddubsw m6, m01327
+ pmaddubsw m7, m01328
+ pmaddwd m6, m31329
+ pmaddwd m7, m31330
+1331
+ packssdw m6, m71332
+ psubw m6, m41333
+ vpermq m6, m8, m61334
+ movu [r2], m61335
+%endmacro1336
+1337
+;-------------------------------------------------------------------------------------------------------------1338
+; void interp_horiz_ps_32xN(const pixel* src, intptr_t srcStride, int16_t* dst, intptr_t dstStride, int coeffIdx, int isRowExt)1339
+;-------------------------------------------------------------------------------------------------------------1340
+%macro IPFILTER_CHROMA_PS_32xN_AVX512 11341
+INIT_ZMM avx5121342
+cglobal interp_4tap_horiz_ps_32x%1, 4,7,91343
+ mov r4d, r4m1344
+ mov r5d, r5m1345
+1346
+%ifdef PIC1347
+ lea r6, [tab_ChromaCoeff]1348
+ vpbroadcastd m0, [r6 + r4 * 4]1349
+%else1350
+ vpbroadcastd m0, [tab_ChromaCoeff + r4 * 4]1351
+%endif1352
+1353
+ vbroadcasti32x8 m1, [interp4_horiz_shuf_load1_avx512]1354
+ vbroadcasti32x8 m2, [interp4_horiz_shuf_load2_avx512]1355
+ vbroadcasti32x8 m3, [pw_1]1356
+ vbroadcasti32x8 m4, [pw_2000]1357
+ mova m8, [interp4_hps_shuf_avx512]1358
+1359
+ ; register map1360
+ ; m0 - interpolate coeff1361
+ ; m1,m2 - load shuffle order table1362
+ ; m3 - constant word 11363
+ ; m4 - constant word 20001364
+ ; m8 - store shuffle order table1365
+1366
+ mov r6d, %11367
+ dec r01368
+ test r5d, r5d1369
+ je .loop1370
+ sub r0, r11371
+ add r6d, 31372
+1373
+.loop:1374
+ PROCESS_IPFILTER_CHROMA_PS_32x1_AVX5121375
+ lea r2, [r2 + 2 * r3]1376
+ lea r0, [r0 + r1]1377
+ dec r6d1378
+ jnz .loop1379
+ RET1380
+%endmacro1381
+1382
+%if ARCH_X86_641383
+ IPFILTER_CHROMA_PS_32xN_AVX512 641384
+ IPFILTER_CHROMA_PS_32xN_AVX512 481385
+ IPFILTER_CHROMA_PS_32xN_AVX512 321386
+ IPFILTER_CHROMA_PS_32xN_AVX512 241387
+ IPFILTER_CHROMA_PS_32xN_AVX512 161388
+ IPFILTER_CHROMA_PS_32xN_AVX512 81389
+%endif1390
+1391
+%macro PROCESS_IPFILTER_CHROMA_PS_16x2_AVX512 01392
+ movu xm6, [r0]1393
+ vinserti32x4 m6, [r0 + 4], 11394
+ vinserti32x4 m6, [r0 + r1], 21395
+ vinserti32x4 m6, [r0 + r1 + 4], 31396
+1397
+ pshufb m7, m6, m21398
+ pshufb m6, m6, m11399
+ pmaddubsw m6, m01400
+ pmaddubsw m7, m01401
+ pmaddwd m6, m31402
+ pmaddwd m7, m31403
+1404
+ packssdw m6, m71405
+ psubw m6, m41406
+ vpermq m6, m8, m61407
+ movu [r2], ym61408
+ vextracti32x8 [r2 + r3], m6, 11409
+%endmacro1410
+1411
+%macro PROCESS_IPFILTER_CHROMA_PS_16x1_AVX512 01412
+ movu xm6, [r0]1413
+ vinserti32x4 m6, [r0 + 4], 11414
+1415
+ pshufb ym7, ym6, ym21416
+ pshufb ym6, ym6, ym11417
+ pmaddubsw ym6, ym01418
+ pmaddubsw ym7, ym01419
+ pmaddwd ym6, ym31420
+ pmaddwd ym7, ym31421
+1422
+ packssdw ym6, ym71423
+ psubw ym6, ym41424
+ vpermq ym6, ym8, ym61425
+ movu [r2], ym61426
+%endmacro1427
+1428
+;-------------------------------------------------------------------------------------------------------------1429
+; void interp_horiz_ps_16xN(const pixel* src, intptr_t srcStride, int16_t* dst, intptr_t dstStride, int coeffIdx, int isRowExt)1430
+;-------------------------------------------------------------------------------------------------------------1431
+%macro IPFILTER_CHROMA_PS_16xN_AVX512 11432
+INIT_ZMM avx5121433
+cglobal interp_4tap_horiz_ps_16x%1, 4,7,91434
+ mov r4d, r4m1435
+ mov r5d, r5m1436
+ add r3, r31437
+1438
+%ifdef PIC1439
+ lea r6, [tab_ChromaCoeff]1440
+ vpbroadcastd m0, [r6 + r4 * 4]1441
+%else1442
+ vpbroadcastd m0, [tab_ChromaCoeff + r4 * 4]1443
+%endif1444
+1445
+ vbroadcasti32x8 m1, [interp4_horiz_shuf_load1_avx512]1446
+ vbroadcasti32x8 m2, [interp4_horiz_shuf_load2_avx512]1447
+ vbroadcasti32x8 m3, [pw_1]1448
+ vbroadcasti32x8 m4, [pw_2000]1449
+ mova m8, [interp4_hps_store_16xN_avx512]1450
+1451
+ ; register map1452
+ ; m0 - interpolate coeff1453
+ ; m1,m2 - load shuffle order table1454
+ ; m3 - constant word 11455
+ ; m4 - constant word 20001456
+ ; m8 - store shuffle order table1457
+1458
+ mov r6d, %11459
+ dec r01460
+ test r5d, r5d1461
+ je .loop1462
+ sub r0, r11463
+ add r6d, 31464
+ PROCESS_IPFILTER_CHROMA_PS_16x1_AVX5121465
+ lea r2, [r2 + r3]1466
+ lea r0, [r0 + r1]1467
+ dec r6d1468
+1469
+.loop:1470
+ PROCESS_IPFILTER_CHROMA_PS_16x2_AVX5121471
+ lea r2, [r2 + 2 * r3]1472
+ lea r0, [r0 + 2 * r1]1473
+ sub r6d, 21474
+ jnz .loop1475
+1476
+ RET1477
+%endmacro1478
+1479
+%if ARCH_X86_64 == 11480
+ IPFILTER_CHROMA_PS_16xN_AVX512 641481
+ IPFILTER_CHROMA_PS_16xN_AVX512 321482
+ IPFILTER_CHROMA_PS_16xN_AVX512 241483
+ IPFILTER_CHROMA_PS_16xN_AVX512 161484
+ IPFILTER_CHROMA_PS_16xN_AVX512 121485
+ IPFILTER_CHROMA_PS_16xN_AVX512 81486
+ IPFILTER_CHROMA_PS_16xN_AVX512 41487
+%endif1488
+1489
+%macro PROCESS_IPFILTER_CHROMA_PS_48x1_AVX512 01490
+ movu ym6, [r0]1491
+ vinserti32x8 m6, [r0 + 4], 11492
+ pshufb m7, m6, m21493
+ pshufb m6, m6, m11494
+ pmaddubsw m6, m01495
+ pmaddubsw m7, m01496
+ pmaddwd m6, m31497
+ pmaddwd m7, m31498
+1499
+ packssdw m6, m71500
+ psubw m6, m41501
+ vpermq m6, m8, m61502
+ movu [r2], m61503
+1504
+ movu xm6, [r0 + 32]1505
+ vinserti32x4 m6, [r0 + 36], 11506
+ pshufb ym7, ym6, ym21507
+ pshufb ym6, ym6, ym11508
+ pmaddubsw ym6, ym01509
+ pmaddubsw ym7, ym01510
+ pmaddwd ym6, ym31511
+ pmaddwd ym7, ym31512
+1513
+ packssdw ym6, ym71514
+ psubw ym6, ym41515
+ vpermq ym6, ym9, ym61516
+ movu [r2 + mmsize],ym61517
+%endmacro1518
+1519
+;-------------------------------------------------------------------------------------------------------------1520
+; void interp_horiz_ps_48xN(const pixel* src, intptr_t srcStride, int16_t* dst, intptr_t dstStride, int coeffIdx, int isRowExt)1521
+;-------------------------------------------------------------------------------------------------------------1522
+%macro IPFILTER_CHROMA_PS_48xN_AVX512 11523
+INIT_ZMM avx5121524
+cglobal interp_4tap_horiz_ps_48x%1, 4,7,101525
+ mov r4d, r4m1526
+ mov r5d, r5m1527
+1528
+%ifdef PIC1529
+ lea r6, [tab_ChromaCoeff]1530
+ vpbroadcastd m0, [r6 + r4 * 4]1531
+%else1532
+ vpbroadcastd m0, [tab_ChromaCoeff + r4 * 4]1533
+%endif1534
+1535
+ vbroadcasti32x8 m1, [interp4_horiz_shuf_load1_avx512]1536
+ vbroadcasti32x8 m2, [interp4_horiz_shuf_load2_avx512]1537
+ vbroadcasti32x8 m3, [pw_1]1538
+ vbroadcasti32x8 m4, [pw_2000]1539
+ mova m8, [interp4_hps_shuf_avx512]1540
+ mova m9, [interp4_hps_store_16xN_avx512]1541
+1542
+ ; register map1543
+ ; m0 - interpolate coeff1544
+ ; m1,m2 - load shuffle order table1545
+ ; m3 - constant word 11546
+ ; m4 - constant word 20001547
+ ; m8 - store shuffle order table1548
+1549
+ mov r6d, %11550
+ dec r01551
+ test r5d, r5d1552
+ je .loop1553
+ sub r0, r11554
+ add r6d, 31555
+1556
+.loop:1557
+ PROCESS_IPFILTER_CHROMA_PS_48x1_AVX5121558
+ lea r2, [r2 + 2 * r3]1559
+ lea r0, [r0 + r1]1560
+ dec r6d1561
+ jnz .loop1562
+ RET1563
+%endmacro1564
+1565
+%if ARCH_X86_64 == 11566
+ IPFILTER_CHROMA_PS_48xN_AVX512 641567
+%endif1568
+1569
+;-------------------------------------------------------------------------------------------------------------1570
+;avx512 chroma_vpp and chroma_vps code start1571
+;-------------------------------------------------------------------------------------------------------------1572
+%macro PROCESS_CHROMA_VERT_16x4_AVX512 11573
+ lea r5, [r0 + 4 * r1]1574
+ movu xm1, [r0]1575
+ movu xm3, [r0 + r1]1576
+ vinserti32x4 m1, [r0 + r1], 11577
+ vinserti32x4 m3, [r0 + 2 * r1], 11578
+ vinserti32x4 m1, [r0 + 2 * r1], 21579
+ vinserti32x4 m3, [r0 + r6], 21580
+ vinserti32x4 m1, [r0 + r6], 31581
+ vinserti32x4 m3, [r0 + 4 * r1], 31582
+1583
+ punpcklbw m0, m1, m31584
+ pmaddubsw m0, m81585
+ punpckhbw m1, m31586
+ pmaddubsw m1, m81587
+1588
+ movu xm4, [r0 + 2 * r1]1589
+ movu xm5, [r0 + r6]1590
+ vinserti32x4 m4, [r0 + r6], 11591
+ vinserti32x4 m5, [r5], 11592
+ vinserti32x4 m4, [r5], 21593
+ vinserti32x4 m5, [r5 + r1], 21594
+ vinserti32x4 m4, [r5 + r1], 31595
+ vinserti32x4 m5, [r5 + 2 * r1], 31596
+1597
+ punpcklbw m3, m4, m51598
+ pmaddubsw m3, m91599
+ punpckhbw m4, m51600
+ pmaddubsw m4, m91601
+1602
+ paddw m0, m31603
+ paddw m1, m41604
+%ifidn %1,pp1605
+ pmulhrsw m0, m71606
+ pmulhrsw m1, m71607
+ packuswb m0, m11608
+ movu [r2], xm01609
+ vextracti32x4 [r2 + r3], m0, 11610
+ vextracti32x4 [r2 + 2 * r3], m0, 21611
+ vextracti32x4 [r2 + r7], m0, 31612
+%else1613
+ psubw m0, m71614
+ psubw m1, m71615
+ mova m2, m101616
+ mova m3, m111617
+1618
+ vpermi2q m2, m0, m11619
+ vpermi2q m3, m0, m11620
+ 1621
+ movu [r2], ym21622
+ vextracti32x8 [r2 + r3], m2, 11623
+ movu [r2 + 2 * r3], ym31624
+ vextracti32x8 [r2 + r7], m3, 11625
+%endif1626
+%endmacro1627
+1628
+;-----------------------------------------------------------------------------------------------------------------1629
+; void interp_4tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)1630
+;-----------------------------------------------------------------------------------------------------------------1631
+%macro FILTER_VERT_CHROMA_16xN_AVX512 21632
+INIT_ZMM avx5121633
+cglobal interp_4tap_vert_%1_16x%2, 4, 10, 121634
+ mov r4d, r4m1635
+ shl r4d, 71636
+ sub r0, r11637
+1638
+%ifdef PIC1639
+ lea r5, [tab_ChromaCoeffVer_32_avx512]1640
+ mova m8, [r5 + r4]1641
+ mova m9, [r5 + r4 + mmsize]1642
+%else1643
+ mova m8, [tab_ChromaCoeffVer_32_avx512 + r4]1644
+ mova m9, [tab_ChromaCoeffVer_32_avx512 + r4 + mmsize]1645
+%endif1646
+1647
+%ifidn %1, pp1648
+ vbroadcasti32x8 m7, [pw_512]1649
+%else1650
+ shl r3d, 11651
+ vbroadcasti32x8 m7, [pw_2000]1652
+ mova m10, [interp4_vps_store1_avx512]1653
+ mova m11, [interp4_vps_store2_avx512]1654
+%endif1655
+ lea r6, [3 * r1]1656
+ lea r7, [3 * r3]1657
+1658
+%rep %2/4 - 11659
+ PROCESS_CHROMA_VERT_16x4_AVX512 %11660
+ lea r0, [r0 + 4 * r1]1661
+ lea r2, [r2 + 4 * r3]1662
+%endrep1663
+ PROCESS_CHROMA_VERT_16x4_AVX512 %11664
+ RET1665
+%endmacro1666
+1667
+%if ARCH_X86_641668
+ FILTER_VERT_CHROMA_16xN_AVX512 pp, 41669
+ FILTER_VERT_CHROMA_16xN_AVX512 pp, 81670
+ FILTER_VERT_CHROMA_16xN_AVX512 pp, 121671
+ FILTER_VERT_CHROMA_16xN_AVX512 pp, 161672
+ FILTER_VERT_CHROMA_16xN_AVX512 pp, 241673
+ FILTER_VERT_CHROMA_16xN_AVX512 pp, 321674
+ FILTER_VERT_CHROMA_16xN_AVX512 pp, 641675
+1676
+ FILTER_VERT_CHROMA_16xN_AVX512 ps, 41677
+ FILTER_VERT_CHROMA_16xN_AVX512 ps, 81678
+ FILTER_VERT_CHROMA_16xN_AVX512 ps, 121679
+ FILTER_VERT_CHROMA_16xN_AVX512 ps, 161680
+ FILTER_VERT_CHROMA_16xN_AVX512 ps, 241681
+ FILTER_VERT_CHROMA_16xN_AVX512 ps, 321682
+ FILTER_VERT_CHROMA_16xN_AVX512 ps, 641683
+%endif1684
+%macro PROCESS_CHROMA_VERT_32x4_AVX512 11685
+ movu ym1, [r0]1686
+ movu ym3, [r0 + r1]1687
+ vinserti32x8 m1, [r0 + 2 * r1], 11688
+ vinserti32x8 m3, [r0 + r6], 11689
+ punpcklbw m0, m1, m31690
+ pmaddubsw m0, m81691
+ punpckhbw m1, m31692
+ pmaddubsw m1, m81693
+1694
+ movu ym4, [r0 + 2 * r1]1695
+ vinserti32x8 m4, [r0 + 4 * r1], 11696
+ punpcklbw m2, m3, m41697
+ pmaddubsw m2, m81698
+ punpckhbw m3, m41699
+ pmaddubsw m3, m81700
+1701
+ lea r0, [r0 + 2 * r1]1702
+1703
+ movu ym5, [r0 + r1]1704
+ vinserti32x8 m5, [r0 + r6], 11705
+ punpcklbw m6, m4, m51706
+ pmaddubsw m6, m91707
+ paddw m0, m61708
+ punpckhbw m4, m51709
+ pmaddubsw m4, m91710
+ paddw m1, m41711
+1712
+ movu ym4, [r0 + 2 * r1]1713
+ vinserti32x8 m4, [r0 + 4 * r1], 11714
+ punpcklbw m6, m5, m41715
+ pmaddubsw m6, m91716
+ paddw m2, m61717
+ punpckhbw m5, m41718
+ pmaddubsw m5, m91719
+ paddw m3, m51720
+1721
+%ifidn %1,pp1722
+ pmulhrsw m0, m71723
+ pmulhrsw m1, m71724
+ pmulhrsw m2, m71725
+ pmulhrsw m3, m71726
+ packuswb m0, m11727
+ packuswb m2, m31728
+ movu [r2], ym01729
+ movu [r2 + r3], ym21730
+ vextracti32x8 [r2 + 2 * r3], m0, 11731
+ vextracti32x8 [r2 + r7], m2, 11732
+%else1733
+ psubw m0, m71734
+ psubw m1, m71735
+ psubw m2, m71736
+ psubw m3, m71737
+1738
+ mova m4, m101739
+ mova m5, m111740
+ vpermi2q m4, m0, m11741
+ vpermi2q m5, m0, m11742
+ mova m6, m101743
+ mova m12, m111744
+ vpermi2q m6, m2, m31745
+ vpermi2q m12, m2, m31746
+1747
+ movu [r2], m41748
+ movu [r2 + r3], m61749
+ movu [r2 + 2 * r3], m51750
+ movu [r2 + r7], m121751
+%endif1752
+%endmacro1753
+1754
+;-----------------------------------------------------------------------------------------------------------------1755
+; void interp_4tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)1756
+;-----------------------------------------------------------------------------------------------------------------1757
+%macro FILTER_VERT_CHROMA_32xN_AVX512 21758
+INIT_ZMM avx5121759
+cglobal interp_4tap_vert_%1_32x%2, 4, 8, 131760
+ mov r4d, r4m1761
+ shl r4d, 71762
+ sub r0, r11763
+1764
+%ifdef PIC1765
+ lea r5, [tab_ChromaCoeffVer_32_avx512]1766
+ mova m8, [r5 + r4]1767
+ mova m9, [r5 + r4 + mmsize]1768
+%else1769
+ mova m8, [tab_ChromaCoeffVer_32_avx512 + r4]1770
+ mova m9, [tab_ChromaCoeffVer_32_avx512 + r4 + mmsize]1771
+%endif1772
+1773
+%ifidn %1,pp1774
+ vbroadcasti32x8 m7, [pw_512]1775
+%else1776
+ shl r3d, 11777
+ vbroadcasti32x8 m7, [pw_2000]1778
+ mova m10, [interp4_vps_store1_avx512]1779
+ mova m11, [interp4_vps_store2_avx512]1780
+%endif1781
+1782
+ lea r6, [3 * r1]1783
+ lea r7, [3 * r3]1784
+1785
+%rep %2/4 - 11786
+ PROCESS_CHROMA_VERT_32x4_AVX512 %11787
+ lea r0, [r0 + 2 * r1]1788
+ lea r2, [r2 + 4 * r3]1789
+%endrep1790
+ PROCESS_CHROMA_VERT_32x4_AVX512 %11791
+ RET1792
+%endmacro1793
+1794
+%if ARCH_X86_641795
+ FILTER_VERT_CHROMA_32xN_AVX512 pp, 81796
+ FILTER_VERT_CHROMA_32xN_AVX512 pp, 161797
+ FILTER_VERT_CHROMA_32xN_AVX512 pp, 241798
+ FILTER_VERT_CHROMA_32xN_AVX512 pp, 321799
+ FILTER_VERT_CHROMA_32xN_AVX512 pp, 481800
+ FILTER_VERT_CHROMA_32xN_AVX512 pp, 641801
+1802
+ FILTER_VERT_CHROMA_32xN_AVX512 ps, 81803
+ FILTER_VERT_CHROMA_32xN_AVX512 ps, 161804
+ FILTER_VERT_CHROMA_32xN_AVX512 ps, 241805
+ FILTER_VERT_CHROMA_32xN_AVX512 ps, 321806
+ FILTER_VERT_CHROMA_32xN_AVX512 ps, 481807
+ FILTER_VERT_CHROMA_32xN_AVX512 ps, 641808
+%endif1809
+%macro PROCESS_CHROMA_VERT_48x4_AVX512 11810
+ movu ym1, [r0]1811
+ movu ym3, [r0 + r1]1812
+ vinserti32x8 m1, [r0 + 2 * r1], 11813
+ vinserti32x8 m3, [r0 + r6], 11814
+ punpcklbw m0, m1, m31815
+ pmaddubsw m0, m81816
+ punpckhbw m1, m31817
+ pmaddubsw m1, m81818
+1819
+ movu ym4, [r0 + 2 * r1]1820
+ vinserti32x8 m4, [r0 + 4 * r1], 11821
+ punpcklbw m2, m3, m41822
+ pmaddubsw m2, m81823
+ punpckhbw m3, m41824
+ pmaddubsw m3, m81825
+1826
+ lea r5, [r0 + 4 * r1]1827
+1828
+ movu ym5, [r0 + r6]1829
+ vinserti32x8 m5, [r5 + r1], 11830
+ punpcklbw m6, m4, m51831
+ pmaddubsw m6, m91832
+ paddw m0, m61833
+ punpckhbw m4, m51834
+ pmaddubsw m4, m91835
+ paddw m1, m41836
+1837
+ movu ym4, [r0 + 4 * r1]1838
+ vinserti32x8 m4, [r5 + 2 * r1], 11839
+ punpcklbw m6, m5, m41840
+ pmaddubsw m6, m91841
+ paddw m2, m61842
+ punpckhbw m5, m41843
+ pmaddubsw m5, m91844
+ paddw m3, m51845
+%ifidn %1, pp1846
+ pmulhrsw m0, m71847
+ pmulhrsw m1, m71848
+ pmulhrsw m2, m71849
+ pmulhrsw m3, m71850
+1851
+ packuswb m0, m11852
+ packuswb m2, m31853
+ movu [r2], ym01854
+ movu [r2 + r3], ym21855
+ vextracti32x8 [r2 + 2 * r3], m0, 11856
+ vextracti32x8 [r2 + r7], m2, 11857
+%else1858
+ psubw m0, m71859
+ psubw m1, m71860
+ psubw m2, m71861
+ psubw m3, m71862
+1863
+ mova m4, m101864
+ mova m5, m111865
+ vpermi2q m4, m0, m11866
+ vpermi2q m5, m0, m11867
+ mova m6, m101868
+ mova m12, m111869
+ vpermi2q m6, m2, m31870
+ vpermi2q m12, m2, m31871
+1872
+ movu [r2], m41873
+ movu [r2 + r3], m61874
+ movu [r2 + 2 * r3], m51875
+ movu [r2 + r7], m121876
+%endif1877
+ movu xm1, [r0 + mmsize/2]1878
+ movu xm3, [r0 + r1 + mmsize/2]1879
+ vinserti32x4 m1, [r0 + r1 + mmsize/2], 11880
+ vinserti32x4 m3, [r0 + 2 * r1 + mmsize/2], 11881
+ vinserti32x4 m1, [r0 + 2 * r1 + mmsize/2], 21882
+ vinserti32x4 m3, [r0 + r6 + mmsize/2], 21883
+ vinserti32x4 m1, [r0 + r6 + mmsize/2], 31884
+ vinserti32x4 m3, [r0 + 4 * r1 + mmsize/2], 31885
+1886
+ punpcklbw m0, m1, m31887
+ pmaddubsw m0, m81888
+ punpckhbw m1, m31889
+ pmaddubsw m1, m81890
+1891
+ movu xm4, [r0 + 2 * r1 + mmsize/2]1892
+ movu xm5, [r0 + r6 + mmsize/2]1893
+ vinserti32x4 m4, [r0 + r6 + mmsize/2], 11894
+ vinserti32x4 m5, [r5 + mmsize/2], 11895
+ vinserti32x4 m4, [r5 + mmsize/2], 21896
+ vinserti32x4 m5, [r5 + r1 + mmsize/2], 21897
+ vinserti32x4 m4, [r5 + r1 + mmsize/2], 31898
+ vinserti32x4 m5, [r5 + 2 * r1 + mmsize/2], 31899
+1900
+ punpcklbw m3, m4, m51901
+ pmaddubsw m3, m91902
+ punpckhbw m4, m51903
+ pmaddubsw m4, m91904
+ paddw m0, m31905
+ paddw m1, m41906
+%ifidn %1, pp1907
+ pmulhrsw m0, m71908
+ pmulhrsw m1, m71909
+ packuswb m0, m11910
+ movu [r2 + mmsize/2], xm01911
+ vextracti32x4 [r2 + r3 + mmsize/2], m0, 11912
+ vextracti32x4 [r2 + 2 * r3 + mmsize/2], m0, 21913
+ vextracti32x4 [r2 + r7 + mmsize/2], m0, 31914
+%else1915
+ psubw m0, m71916
+ psubw m1, m71917
+ mova m2, m101918
+ mova m3, m111919
+1920
+ vpermi2q m2, m0, m11921
+ vpermi2q m3, m0, m11922
+1923
+ movu [r2 + mmsize], ym21924
+ vextracti32x8 [r2 + r3 + mmsize], m2, 11925
+ movu [r2 + 2 * r3 + mmsize], ym31926
+ vextracti32x8 [r2 + r7 + mmsize], m3, 11927
+%endif1928
+%endmacro1929
+;-----------------------------------------------------------------------------------------------------------------1930
+; void interp_8tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)1931
+;-----------------------------------------------------------------------------------------------------------------1932
+%macro FILTER_VERT_CHROMA_48x64_AVX512 11933
+INIT_ZMM avx5121934
+cglobal interp_4tap_vert_%1_48x64, 4, 8, 131935
+ mov r4d, r4m1936
+ shl r4d, 71937
+ sub r0, r11938
+1939
+%ifdef PIC1940
+ lea r5, [tab_ChromaCoeffVer_32_avx512]1941
+ mova m8, [r5 + r4]1942
+ mova m9, [r5 + r4 + mmsize]1943
+%else1944
+ mova m8, [tab_ChromaCoeffVer_32_avx512 + r4]1945
+ mova m9, [tab_ChromaCoeffVer_32_avx512 + r4 + mmsize]1946
+%endif1947
+1948
+%ifidn %1, pp1949
+ vbroadcasti32x8 m7, [pw_512]1950
+%else1951
+ shl r3d, 11952
+ vbroadcasti32x8 m7, [pw_2000]1953
+ mova m10, [interp4_vps_store1_avx512]1954
+ mova m11, [interp4_vps_store2_avx512]1955
+%endif1956
+1957
+ lea r6, [3 * r1]1958
+ lea r7, [3 * r3]1959
+%rep 151960
+ PROCESS_CHROMA_VERT_48x4_AVX512 %11961
+ lea r0, [r0 + 4 * r1]1962
+ lea r2, [r2 + 4 * r3]1963
+%endrep1964
+ PROCESS_CHROMA_VERT_48x4_AVX512 %11965
+ RET1966
+%endmacro1967
+1968
+%if ARCH_X86_641969
+ FILTER_VERT_CHROMA_48x64_AVX512 pp1970
+ FILTER_VERT_CHROMA_48x64_AVX512 ps1971
+%endif1972
+%macro PROCESS_CHROMA_VERT_64x4_AVX512 11973
+ movu m0, [r0] ; m0 = row 01974
+ movu m1, [r0 + r1] ; m1 = row 11975
+ punpcklbw m2, m0, m11976
+ punpckhbw m3, m0, m11977
+ pmaddubsw m2, m101978
+ pmaddubsw m3, m101979
+ movu m0, [r0 + r1 * 2] ; m0 = row 21980
+ punpcklbw m4, m1, m01981
+ punpckhbw m5, m1, m01982
+ pmaddubsw m4, m101983
+ pmaddubsw m5, m101984
+ movu m1, [r0 + r4] ; m1 = row 31985
+ punpcklbw m6, m0, m11986
+ punpckhbw m7, m0, m11987
+ pmaddubsw m8, m6, m111988
+ pmaddubsw m9, m7, m111989
+ pmaddubsw m6, m101990
+ pmaddubsw m7, m101991
+ paddw m2, m81992
+ paddw m3, m91993
+1994
+%ifidn %1,pp1995
+ pmulhrsw m2, m121996
+ pmulhrsw m3, m121997
+ packuswb m2, m31998
+ movu [r2], m21999
+%else2000
+ psubw m2, m122001
+ psubw m3, m122002
+ movu m8, m132003
+ movu m9, m142004
+ vpermi2q m8, m2, m32005
+ vpermi2q m9, m2, m32006
+ movu [r2], m82007
+ movu [r2 + mmsize], m92008
+%endif2009
+2010
+ lea r0, [r0 + r1 * 4]2011
+ movu m0, [r0] ; m0 = row 42012
+ punpcklbw m2, m1, m02013
+ punpckhbw m3, m1, m02014
+ pmaddubsw m8, m2, m112015
+ pmaddubsw m9, m3, m112016
+ pmaddubsw m2, m102017
+ pmaddubsw m3, m102018
+ paddw m4, m82019
+ paddw m5, m92020
+2021
+%ifidn %1,pp2022
+ pmulhrsw m4, m122023
+ pmulhrsw m5, m122024
+ packuswb m4, m52025
+ movu [r2 + r3], m42026
+%else2027
+ psubw m4, m122028
+ psubw m5, m122029
+ movu m8, m132030
+ movu m9, m142031
+ vpermi2q m8, m4, m52032
+ vpermi2q m9, m4, m52033
+ movu [r2 + r3], m82034
+ movu [r2 + r3 + mmsize], m92035
+%endif2036
+2037
+ movu m1, [r0 + r1] ; m1 = row 52038
+ punpcklbw m4, m0, m12039
+ punpckhbw m5, m0, m12040
+ pmaddubsw m4, m112041
+ pmaddubsw m5, m112042
+ paddw m6, m42043
+ paddw m7, m52044
+2045
+%ifidn %1,pp2046
+ pmulhrsw m6, m122047
+ pmulhrsw m7, m122048
+ packuswb m6, m72049
+ movu [r2 + r3 * 2], m62050
+%else2051
+ psubw m6, m122052
+ psubw m7, m122053
+ movu m8, m132054
+ movu m9, m142055
+ vpermi2q m8, m6, m72056
+ vpermi2q m9, m6, m72057
+ movu [r2 + 2 * r3], m82058
+ movu [r2 + 2 * r3 + mmsize], m92059
+%endif2060
+ movu m0, [r0 + r1 * 2] ; m0 = row 62061
+ punpcklbw m6, m1, m02062
+ punpckhbw m7, m1, m02063
+ pmaddubsw m6, m112064
+ pmaddubsw m7, m112065
+ paddw m2, m62066
+ paddw m3, m72067
+2068
+%ifidn %1,pp2069
+ pmulhrsw m2, m122070
+ pmulhrsw m3, m122071
+ packuswb m2, m32072
+ movu [r2 + r5], m22073
+%else2074
+ psubw m2, m122075
+ psubw m3, m122076
+ movu m8, m132077
+ movu m9, m142078
+ vpermi2q m8, m2, m32079
+ vpermi2q m9, m2, m32080
+ movu [r2 + r5], m82081
+ movu [r2 + r5 + mmsize], m92082
+%endif2083
+%endmacro2084
+2085
+%macro FILTER_VER_CHROMA_AVX512_64xN 22086
+INIT_ZMM avx5122087
+cglobal interp_4tap_vert_%1_64x%2, 4, 6, 152088
+ mov r4d, r4m2089
+ shl r4d, 72090
+2091
+%ifdef PIC2092
+ lea r5, [tab_ChromaCoeffVer_32_avx512]2093
+ mova m10, [r5 + r4]2094
+ mova m11, [r5 + r4 + mmsize]2095
+%else2096
+ mova m10, [tab_ChromaCoeffVer_32_avx512 + r4]2097
+ mova m11, [tab_ChromaCoeffVer_32_avx512 + r4 + mmsize]2098
+%endif2099
+2100
+%ifidn %1,pp2101
+ vbroadcasti32x8 m12, [pw_512]2102
+%else2103
+ shl r3d, 12104
+ vbroadcasti32x8 m12, [pw_2000]2105
+ mova m13, [interp4_vps_store1_avx512]2106
+ mova m14, [interp4_vps_store2_avx512]2107
+%endif2108
+ lea r4, [r1 * 3]2109
+ sub r0, r12110
+ lea r5, [r3 * 3]2111
+2112
+%rep %2/4 - 12113
+ PROCESS_CHROMA_VERT_64x4_AVX512 %12114
+ lea r2, [r2 + r3 * 4]2115
+%endrep2116
+ PROCESS_CHROMA_VERT_64x4_AVX512 %12117
+ RET2118
+%endmacro2119
+2120
+%if ARCH_X86_64 == 12121
+FILTER_VER_CHROMA_AVX512_64xN pp, 642122
+FILTER_VER_CHROMA_AVX512_64xN pp, 482123
+FILTER_VER_CHROMA_AVX512_64xN pp, 322124
+FILTER_VER_CHROMA_AVX512_64xN pp, 162125
+2126
+FILTER_VER_CHROMA_AVX512_64xN ps, 642127
+FILTER_VER_CHROMA_AVX512_64xN ps, 482128
+FILTER_VER_CHROMA_AVX512_64xN ps, 322129
+FILTER_VER_CHROMA_AVX512_64xN ps, 162130
+%endif2131
+;-------------------------------------------------------------------------------------------------------------2132
+;avx512 chroma_vpp and chroma_vps code end2133
+;-------------------------------------------------------------------------------------------------------------2134
+;-------------------------------------------------------------------------------------------------------------2135
+;avx512 chroma_vss code start2136
+;-------------------------------------------------------------------------------------------------------------2137
+%macro PROCESS_CHROMA_VERT_SS_8x4_AVX512 02138
+ lea r5, [r0 + 4 * r1]2139
+ movu xm1, [r0]2140
+ movu xm3, [r0 + r1]2141
+ vinserti32x4 m1, [r0 + r1], 12142
+ vinserti32x4 m3, [r0 + 2 * r1], 12143
+ vinserti32x4 m1, [r0 + 2 * r1], 22144
+ vinserti32x4 m3, [r0 + r6], 22145
+ vinserti32x4 m1, [r0 + r6], 32146
+ vinserti32x4 m3, [r0 + 4 * r1], 32147
+2148
+ punpcklwd m0, m1, m32149
+ pmaddwd m0, m82150
+ punpckhwd m1, m32151
+ pmaddwd m1, m82152
+2153
+ movu xm4, [r0 + 2 * r1]2154
+ movu xm5, [r0 + r6]2155
+ vinserti32x4 m4, [r0 + r6], 12156
+ vinserti32x4 m5, [r5], 12157
+ vinserti32x4 m4, [r5], 22158
+ vinserti32x4 m5, [r5 + r1], 22159
+ vinserti32x4 m4, [r5 + r1], 32160
+ vinserti32x4 m5, [r5 + 2 * r1], 32161
+2162
+ punpcklwd m3, m4, m52163
+ pmaddwd m3, m92164
+ punpckhwd m4, m52165
+ pmaddwd m4, m92166
+2167
+ paddd m0, m32168
+ paddd m1, m42169
+2170
+ psrad m0, 62171
+ psrad m1, 62172
+ packssdw m0, m12173
+ movu [r2], xm02174
+ vextracti32x4 [r2 + r3], m0, 12175
+ vextracti32x4 [r2 + 2 * r3], m0, 22176
+ vextracti32x4 [r2 + r7], m0, 32177
+%endmacro2178
+2179
+;-----------------------------------------------------------------------------------------------------------------2180
+; void interp_4tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)2181
+;-----------------------------------------------------------------------------------------------------------------2182
+%macro FILTER_VER_SS_CHROMA_8xN_AVX512 12183
+INIT_ZMM avx5122184
+cglobal interp_4tap_vert_ss_8x%1, 5, 8, 102185
+ add r1d, r1d2186
+ add r3d, r3d2187
+ sub r0, r12188
+ shl r4d, 72189
+%ifdef PIC2190
+ lea r5, [pw_ChromaCoeffVer_32_avx512]2191
+ mova m8, [r5 + r4]2192
+ mova m9, [r5 + r4 + mmsize]2193
+%else2194
+ lea r5, [pw_ChromaCoeffVer_32_avx512 + r4]2195
+ mova m8, [r5]2196
+ mova m9, [r5 + mmsize]2197
+%endif2198
+ lea r6, [3 * r1]2199
+ lea r7, [3 * r3]2200
+2201
+%rep %1/4 - 12202
+ PROCESS_CHROMA_VERT_SS_8x4_AVX5122203
+ lea r0, [r0 + 4 * r1]2204
+ lea r2, [r2 + 4 * r3]2205
+%endrep2206
+ PROCESS_CHROMA_VERT_SS_8x4_AVX5122207
+ RET2208
+%endmacro2209
+2210
+%if ARCH_X86_642211
+ FILTER_VER_SS_CHROMA_8xN_AVX512 42212
+ FILTER_VER_SS_CHROMA_8xN_AVX512 82213
+ FILTER_VER_SS_CHROMA_8xN_AVX512 122214
+ FILTER_VER_SS_CHROMA_8xN_AVX512 162215
+ FILTER_VER_SS_CHROMA_8xN_AVX512 322216
+ FILTER_VER_SS_CHROMA_8xN_AVX512 642217
+%endif2218
+2219
+%macro PROCESS_CHROMA_VERT_S_16x4_AVX512 12220
+ movu ym1, [r0]2221
+ lea r6, [r0 + 2 * r1]2222
+ vinserti32x8 m1, [r6], 12223
+ movu ym3, [r0 + r1]2224
+ vinserti32x8 m3, [r6 + r1], 12225
+ punpcklwd m0, m1, m32226
+ pmaddwd m0, m72227
+ punpckhwd m1, m32228
+ pmaddwd m1, m72229
+2230
+ movu ym4, [r0 + 2 * r1]2231
+ vinserti32x8 m4, [r6 + 2 * r1], 12232
+ punpcklwd m2, m3, m42233
+ pmaddwd m2, m72234
+ punpckhwd m3, m42235
+ pmaddwd m3, m72236
+2237
+ movu ym5, [r0 + r4]2238
+ vinserti32x8 m5, [r6 + r4], 12239
+ punpcklwd m6, m4, m52240
+ pmaddwd m6, m82241
+ paddd m0, m62242
+ punpckhwd m4, m52243
+ pmaddwd m4, m82244
+ paddd m1, m42245
+2246
+ movu ym4, [r0 + 4 * r1]2247
+ vinserti32x8 m4, [r6 + 4 * r1], 12248
+ punpcklwd m6, m5, m42249
+ pmaddwd m6, m82250
+ paddd m2, m62251
+ punpckhwd m5, m42252
+ pmaddwd m5, m82253
+ paddd m3, m52254
+2255
+%ifidn %1, sp2256
+ paddd m0, m92257
+ paddd m1, m92258
+ paddd m2, m92259
+ paddd m3, m92260
+2261
+ psrad m0, 122262
+ psrad m1, 122263
+ psrad m2, 122264
+ psrad m3, 122265
+2266
+ packssdw m0, m12267
+ packssdw m2, m32268
+ packuswb m0, m22269
+ vpermq m0, m10, m02270
+ movu [r2], xm02271
+ vextracti32x4 [r2 + r3], m0, 22272
+ vextracti32x4 [r2 + 2 * r3], m0, 12273
+ vextracti32x4 [r2 + r5], m0, 32274
+%else2275
+ psrad m0, 62276
+ psrad m1, 62277
+ psrad m2, 62278
+ psrad m3, 62279
+ packssdw m0, m12280
+ packssdw m2, m32281
+2282
+ movu [r2], ym02283
+ movu [r2 + r3], ym22284
+ vextracti32x8 [r2 + 2 * r3], m0, 12285
+ vextracti32x8 [r2 + r5], m2, 12286
+%endif2287
+%endmacro2288
+2289
+;-----------------------------------------------------------------------------------------------------------------2290
+; void interp_4tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)2291
+;-----------------------------------------------------------------------------------------------------------------2292
+%macro FILTER_VER_S_CHROMA_16xN_AVX512 22293
+INIT_ZMM avx5122294
+cglobal interp_4tap_vert_%1_16x%2, 4, 7, 112295
+ mov r4d, r4m2296
+ shl r4d, 72297
+2298
+%ifdef PIC2299
+ lea r5, [pw_ChromaCoeffVer_32_avx512]2300
+ mova m7, [r5 + r4]2301
+ mova m8, [r5 + r4 + mmsize]2302
+%else2303
+ mova m7, [pw_ChromaCoeffVer_32_avx512 + r4]2304
+ mova m8, [pw_ChromaCoeffVer_32_avx512 + r4 + mmsize]2305
+%endif2306
+2307
+%ifidn %1, sp2308
+ vbroadcasti32x4 m9, [pd_526336]2309
+ mova m10, [interp8_vsp_store_avx512]2310
+%else2311
+ add r3d, r3d2312
+%endif2313
+ add r1d, r1d2314
+ sub r0, r12315
+ lea r4, [r1 * 3]2316
+ lea r5, [r3 * 3]2317
+2318
+%rep %2/4 - 12319
+ PROCESS_CHROMA_VERT_S_16x4_AVX512 %12320
+ lea r0, [r0 + 4 * r1]2321
+ lea r2, [r2 + 4 * r3]2322
+%endrep2323
+ PROCESS_CHROMA_VERT_S_16x4_AVX512 %12324
+ RET2325
+%endmacro2326
+2327
+%if ARCH_X86_642328
+ FILTER_VER_S_CHROMA_16xN_AVX512 ss, 42329
+ FILTER_VER_S_CHROMA_16xN_AVX512 ss, 82330
+ FILTER_VER_S_CHROMA_16xN_AVX512 ss, 122331
+ FILTER_VER_S_CHROMA_16xN_AVX512 ss, 162332
+ FILTER_VER_S_CHROMA_16xN_AVX512 ss, 242333
+ FILTER_VER_S_CHROMA_16xN_AVX512 ss, 322334
+ FILTER_VER_S_CHROMA_16xN_AVX512 ss, 642335
+ FILTER_VER_S_CHROMA_16xN_AVX512 sp, 42336
+ FILTER_VER_S_CHROMA_16xN_AVX512 sp, 82337
+ FILTER_VER_S_CHROMA_16xN_AVX512 sp, 122338
+ FILTER_VER_S_CHROMA_16xN_AVX512 sp, 162339
+ FILTER_VER_S_CHROMA_16xN_AVX512 sp, 242340
+ FILTER_VER_S_CHROMA_16xN_AVX512 sp, 322341
+ FILTER_VER_S_CHROMA_16xN_AVX512 sp, 642342
+%endif2343
+2344
+%macro PROCESS_CHROMA_VERT_SS_24x8_AVX512 02345
+ movu ym1, [r0]2346
+ lea r6, [r0 + 2 * r1]2347
+ lea r8, [r0 + 4 * r1]2348
+ lea r9, [r8 + 2 * r1]2349
+2350
+ movu ym10, [r8]2351
+ movu ym3, [r0 + r1]2352
+ movu ym12, [r8 + r1]2353
+ vinserti32x8 m1, [r6], 12354
+ vinserti32x8 m10, [r9], 12355
+ vinserti32x8 m3, [r6 + r1], 12356
+ vinserti32x8 m12, [r9 + r1], 12357
+2358
+ punpcklwd m0, m1, m32359
+ punpcklwd m9, m10, m122360
+ pmaddwd m0, m162361
+ pmaddwd m9, m162362
+ punpckhwd m1, m32363
+ punpckhwd m10, m122364
+ pmaddwd m1, m162365
+ pmaddwd m10, m162366
+2367
+ movu ym4, [r0 + 2 * r1]2368
+ movu ym13, [r8 + 2 * r1]2369
+ vinserti32x8 m4, [r6 + 2 * r1], 12370
+ vinserti32x8 m13, [r9 + 2 * r1], 12371
+ punpcklwd m2, m3, m42372
+ punpcklwd m11, m12, m132373
+ pmaddwd m2, m162374
+ pmaddwd m11, m162375
+ punpckhwd m3, m42376
+ punpckhwd m12, m132377
+ pmaddwd m3, m162378
+ pmaddwd m12, m162379
+2380
+ movu ym5, [r0 + r10]2381
+ vinserti32x8 m5, [r6 + r10], 12382
+ movu ym14, [r8 + r10]2383
+ vinserti32x8 m14, [r9 + r10], 12384
+ punpcklwd m6, m4, m52385
+ punpcklwd m15, m13, m142386
+ pmaddwd m6, m172387
+ pmaddwd m15, m172388
+ paddd m0, m62389
+ paddd m9, m152390
+ punpckhwd m4, m52391
+ punpckhwd m13, m142392
+ pmaddwd m4, m172393
+ pmaddwd m13, m172394
+ paddd m1, m42395
+ paddd m10, m132396
+2397
+ movu ym4, [r0 + 4 * r1]2398
+ vinserti32x8 m4, [r6 + 4 * r1], 12399
+ movu ym13, [r8 + 4 * r1]2400
+ vinserti32x8 m13, [r9 + 4 * r1], 12401
+ punpcklwd m6, m5, m42402
+ punpcklwd m15, m14, m132403
+ pmaddwd m6, m172404
+ pmaddwd m15, m172405
+ paddd m2, m62406
+ paddd m11, m152407
+ punpckhwd m5, m42408
+ punpckhwd m14, m132409
+ pmaddwd m5, m172410
+ pmaddwd m14, m172411
+ paddd m3, m52412
+ paddd m12, m142413
+2414
+ psrad m0, 62415
+ psrad m1, 62416
+ psrad m2, 62417
+ psrad m3, 62418
+ psrad m9, 62419
+ psrad m10, 62420
+ psrad m11, 62421
+ psrad m12, 62422
+2423
+ packssdw m0, m12424
+ packssdw m2, m32425
+ packssdw m9, m102426
+ packssdw m11, m122427
+2428
+ movu [r2], ym02429
+ movu [r2 + r3], ym22430
+ vextracti32x8 [r2 + 2 * r3], m0, 12431
+ vextracti32x8 [r2 + r7], m2, 12432
+ lea r11, [r2 + 4 * r3]2433
+ movu [r11], ym92434
+ movu [r11 + r3], ym112435
+ vextracti32x8 [r11 + 2 * r3], m9, 12436
+ vextracti32x8 [r11 + r7], m11, 12437
+2438
+ movu xm1, [r0 + mmsize/2]2439
+ vinserti32x4 m1, [r6 + mmsize/2], 12440
+ vinserti32x4 m1, [r8 + mmsize/2], 22441
+ vinserti32x4 m1, [r9 + mmsize/2], 32442
+ movu xm3, [r0 + r1 + mmsize/2]2443
+ vinserti32x4 m3, [r6 + r1 + mmsize/2], 12444
+ vinserti32x4 m3, [r8 + r1 + mmsize/2], 22445
+ vinserti32x4 m3, [r9 + r1 + mmsize/2], 32446
+ punpcklwd m0, m1, m32447
+ pmaddwd m0, m162448
+ punpckhwd m1, m32449
+ pmaddwd m1, m162450
+2451
+ movu xm4, [r0 + 2 * r1 + mmsize/2]2452
+ vinserti32x4 m4, [r6 + 2 * r1 + mmsize/2], 12453
+ vinserti32x4 m4, [r8 + 2 * r1 + mmsize/2], 22454
+ vinserti32x4 m4, [r9 + 2 * r1 + mmsize/2], 32455
+ punpcklwd m2, m3, m42456
+ pmaddwd m2, m162457
+ punpckhwd m3, m42458
+ pmaddwd m3, m162459
+2460
+ movu xm5, [r0 + r10 + mmsize/2]2461
+ vinserti32x4 m5, [r6 + r10 + mmsize/2], 12462
+ vinserti32x4 m5, [r8 + r10 + mmsize/2], 22463
+ vinserti32x4 m5, [r9 + r10 + mmsize/2], 32464
+ punpcklwd m6, m4, m52465
+ pmaddwd m6, m172466
+ paddd m0, m62467
+ punpckhwd m4, m52468
+ pmaddwd m4, m172469
+ paddd m1, m42470
+2471
+ movu xm4, [r0 + 4 * r1 + mmsize/2]2472
+ vinserti32x4 m4, [r6 + 4 * r1 + mmsize/2], 12473
+ vinserti32x4 m4, [r8 + 4 * r1 + mmsize/2], 22474
+ vinserti32x4 m4, [r9 + 4 * r1 + mmsize/2], 32475
+ punpcklwd m6, m5, m42476
+ pmaddwd m6, m172477
+ paddd m2, m62478
+ punpckhwd m5, m42479
+ pmaddwd m5, m172480
+ paddd m3, m52481
+2482
+ psrad m0, 62483
+ psrad m1, 62484
+ psrad m2, 62485
+ psrad m3, 62486
+2487
+ packssdw m0, m12488
+ packssdw m2, m32489
+2490
+ movu [r2 + mmsize/2], xm02491
+ movu [r2 + r3 + mmsize/2], xm22492
+ vextracti32x4 [r2 + 2 * r3 + mmsize/2], m0, 12493
+ vextracti32x4 [r2 + r7 + mmsize/2], m2, 12494
+ lea r2, [r2 + 4 * r3]2495
+ vextracti32x4 [r2 + mmsize/2], m0, 22496
+ vextracti32x4 [r2 + r3 + mmsize/2], m2, 22497
+ vextracti32x4 [r2 + 2 * r3 + mmsize/2], m0, 32498
+ vextracti32x4 [r2 + r7 + mmsize/2], m2, 32499
+%endmacro2500
+2501
+%macro FILTER_VER_SS_CHROMA_24xN_AVX512 12502
+INIT_ZMM avx5122503
+cglobal interp_4tap_vert_ss_24x%1, 5, 12, 182504
+ add r1d, r1d2505
+ add r3d, r3d2506
+ sub r0, r12507
+ shl r4d, 72508
+%ifdef PIC2509
+ lea r5, [pw_ChromaCoeffVer_32_avx512]2510
+ mova m16, [r5 + r4]2511
+ mova m17, [r5 + r4 + mmsize]2512
+%else2513
+ lea r5, [pw_ChromaCoeffVer_32_avx512 + r4]2514
+ mova m16, [r5]2515
+ mova m17, [r5 + mmsize]2516
+%endif2517
+ lea r10, [3 * r1]2518
+ lea r7, [3 * r3]2519
+%rep %1/8 - 12520
+ PROCESS_CHROMA_VERT_SS_24x8_AVX5122521
+ lea r0, [r8 + 4 * r1]2522
+ lea r2, [r2 + 4 * r3]2523
+%endrep2524
+ PROCESS_CHROMA_VERT_SS_24x8_AVX5122525
+ RET2526
+%endmacro2527
+2528
+%if ARCH_X86_642529
+ FILTER_VER_SS_CHROMA_24xN_AVX512 322530
+ FILTER_VER_SS_CHROMA_24xN_AVX512 642531
+%endif2532
+%macro PROCESS_CHROMA_VERT_S_32x2_AVX512 12533
+ movu m1, [r0]2534
+ movu m3, [r0 + r1]2535
+ punpcklwd m0, m1, m32536
+ pmaddwd m0, m72537
+ punpckhwd m1, m32538
+ pmaddwd m1, m72539
+ movu m4, [r0 + 2 * r1]2540
+ punpcklwd m2, m3, m42541
+ pmaddwd m2, m72542
+ punpckhwd m3, m42543
+ pmaddwd m3, m72544
+ movu m5, [r0 + r4]2545
+ punpcklwd m6, m4, m52546
+ pmaddwd m6, m82547
+ paddd m0, m62548
+ punpckhwd m4, m52549
+ pmaddwd m4, m82550
+ paddd m1, m42551
+ movu m4, [r0 + 4 * r1]2552
+ punpcklwd m6, m5, m42553
+ pmaddwd m6, m82554
+ paddd m2, m62555
+ punpckhwd m5, m42556
+ pmaddwd m5, m82557
+ paddd m3, m52558
+%ifidn %1, sp2559
+ paddd m0, m92560
+ paddd m1, m92561
+ paddd m2, m92562
+ paddd m3, m92563
+2564
+ psrad m0, 122565
+ psrad m1, 122566
+ psrad m2, 122567
+ psrad m3, 122568
+2569
+ packssdw m0, m12570
+ packssdw m2, m32571
+ packuswb m0, m22572
+ vpermq m0, m10, m02573
+ movu [r2], ym02574
+ vextracti32x8 [r2 + r3], m0, 12575
+%else2576
+ psrad m0, 62577
+ psrad m1, 62578
+ psrad m2, 62579
+ psrad m3, 62580
+ packssdw m0, m12581
+ packssdw m2, m32582
+ movu [r2], m02583
+ movu [r2 + r3], m22584
+%endif2585
+%endmacro2586
+2587
+%macro FILTER_VER_S_CHROMA_32xN_AVX512 22588
+INIT_ZMM avx5122589
+cglobal interp_4tap_vert_%1_32x%2, 4, 6, 112590
+ mov r4d, r4m2591
+ shl r4d, 72592
+%ifdef PIC2593
+ lea r5, [pw_ChromaCoeffVer_32_avx512]2594
+ mova m7, [r5 + r4]2595
+ mova m8, [r5 + r4 + mmsize]2596
+%else2597
+ mova m7, [pw_ChromaCoeffVer_32_avx512 + r4]2598
+ mova m8, [pw_ChromaCoeffVer_32_avx512 + r4 + mmsize]2599
+%endif2600
+%ifidn %1, sp2601
+ vbroadcasti32x4 m9, [pd_526336]2602
+ mova m10, [interp8_vsp_store_avx512]2603
+%else2604
+ add r3d, r3d2605
+%endif2606
+ add r1d, r1d2607
+ sub r0, r12608
+ lea r4, [r1 * 3]2609
+ lea r5, [r3 * 3]2610
+%rep %2/2 - 12611
+ PROCESS_CHROMA_VERT_S_32x2_AVX512 %12612
+ lea r0, [r0 + r1 * 2]2613
+ lea r2, [r2 + r3 * 2]2614
+%endrep2615
+ PROCESS_CHROMA_VERT_S_32x2_AVX512 %12616
+ RET2617
+%endmacro2618
+2619
+%if ARCH_X86_642620
+ FILTER_VER_S_CHROMA_32xN_AVX512 ss, 82621
+ FILTER_VER_S_CHROMA_32xN_AVX512 ss, 162622
+ FILTER_VER_S_CHROMA_32xN_AVX512 ss, 242623
+ FILTER_VER_S_CHROMA_32xN_AVX512 ss, 322624
+ FILTER_VER_S_CHROMA_32xN_AVX512 ss, 482625
+ FILTER_VER_S_CHROMA_32xN_AVX512 ss, 642626
+ FILTER_VER_S_CHROMA_32xN_AVX512 sp, 82627
+ FILTER_VER_S_CHROMA_32xN_AVX512 sp, 162628
+ FILTER_VER_S_CHROMA_32xN_AVX512 sp, 242629
+ FILTER_VER_S_CHROMA_32xN_AVX512 sp, 322630
+ FILTER_VER_S_CHROMA_32xN_AVX512 sp, 482631
+ FILTER_VER_S_CHROMA_32xN_AVX512 sp, 642632
+%endif2633
+2634
+%macro PROCESS_CHROMA_VERT_S_48x4_AVX512 12635
+ PROCESS_CHROMA_VERT_S_32x2_AVX512 %12636
+ lea r6, [r0 + 2 * r1]2637
+2638
+ movu m1, [r6]2639
+ movu m3, [r6 + r1]2640
+ punpcklwd m0, m1, m32641
+ pmaddwd m0, m72642
+ punpckhwd m1, m32643
+ pmaddwd m1, m72644
+ movu m4, [r6 + 2 * r1]2645
+ punpcklwd m2, m3, m42646
+ pmaddwd m2, m72647
+ punpckhwd m3, m42648
+ pmaddwd m3, m72649
+2650
+ movu m5, [r6 + r4]2651
+ punpcklwd m6, m4, m52652
+ pmaddwd m6, m82653
+ paddd m0, m62654
+ punpckhwd m4, m52655
+ pmaddwd m4, m82656
+ paddd m1, m42657
+2658
+ movu m4, [r6 + 4 * r1]2659
+ punpcklwd m6, m5, m42660
+ pmaddwd m6, m82661
+ paddd m2, m62662
+ punpckhwd m5, m42663
+ pmaddwd m5, m82664
+ paddd m3, m52665
+2666
+%ifidn %1, sp2667
+ paddd m0, m92668
+ paddd m1, m92669
+ paddd m2, m92670
+ paddd m3, m92671
+2672
+ psrad m0, 122673
+ psrad m1, 122674
+ psrad m2, 122675
+ psrad m3, 122676
+2677
+ packssdw m0, m12678
+ packssdw m2, m32679
+ packuswb m0, m22680
+ vpermq m0, m10, m02681
+ movu [r2 + 2 * r3], ym02682
+ vextracti32x8 [r2 + r5], m0, 12683
+%else2684
+ psrad m0, 62685
+ psrad m1, 62686
+ psrad m2, 62687
+ psrad m3, 62688
+2689
+ packssdw m0, m12690
+ packssdw m2, m32691
+ movu [r2 + 2 * r3], m02692
+ movu [r2 + r5], m22693
+%endif2694
+2695
+ movu ym1, [r0 + mmsize]2696
+ vinserti32x8 m1, [r6 + mmsize], 12697
+ movu ym3, [r0 + r1 + mmsize]2698
+ vinserti32x8 m3, [r6 + r1 + mmsize], 12699
+ punpcklwd m0, m1, m32700
+ pmaddwd m0, m72701
+ punpckhwd m1, m32702
+ pmaddwd m1, m72703
+2704
+ movu ym4, [r0 + 2 * r1 + mmsize]2705
+ vinserti32x8 m4, [r6 + 2 * r1 + mmsize], 12706
+ punpcklwd m2, m3, m42707
+ pmaddwd m2, m72708
+ punpckhwd m3, m42709
+ pmaddwd m3, m72710
+2711
+ movu ym5, [r0 + r4 + mmsize]2712
+ vinserti32x8 m5, [r6 + r4 + mmsize], 12713
+ punpcklwd m6, m4, m52714
+ pmaddwd m6, m82715
+ paddd m0, m62716
+ punpckhwd m4, m52717
+ pmaddwd m4, m82718
+ paddd m1, m42719
+2720
+ movu ym4, [r0 + 4 * r1 + mmsize]2721
+ vinserti32x8 m4, [r6 + 4 * r1 + mmsize], 12722
+ punpcklwd m6, m5, m42723
+ pmaddwd m6, m82724
+ paddd m2, m62725
+ punpckhwd m5, m42726
+ pmaddwd m5, m82727
+ paddd m3, m52728
+2729
+%ifidn %1, sp2730
+ paddd m0, m92731
+ paddd m1, m92732
+ paddd m2, m92733
+ paddd m3, m92734
+2735
+ psrad m0, 122736
+ psrad m1, 122737
+ psrad m2, 122738
+ psrad m3, 122739
+2740
+ packssdw m0, m12741
+ packssdw m2, m32742
+ packuswb m0, m22743
+ vpermq m0, m10, m02744
+ movu [r2 + mmsize/2], xm02745
+ vextracti32x4 [r2 + r3 + mmsize/2], m0, 22746
+ vextracti32x4 [r2 + 2 * r3 + mmsize/2], m0, 12747
+ vextracti32x4 [r2 + r5 + mmsize/2], m0, 32748
+%else2749
+ psrad m0, 62750
+ psrad m1, 62751
+ psrad m2, 62752
+ psrad m3, 62753
+ packssdw m0, m12754
+ packssdw m2, m32755
+2756
+ movu [r2 + mmsize], ym02757
+ movu [r2 + r3 + mmsize], ym22758
+ vextracti32x8 [r2 + 2 * r3 + mmsize], m0, 12759
+ vextracti32x8 [r2 + r5 + mmsize], m2, 12760
+%endif2761
+%endmacro2762
+2763
+%macro FILTER_VER_S_CHROMA_48x64_AVX512 12764
+INIT_ZMM avx5122765
+cglobal interp_4tap_vert_%1_48x64, 4, 7, 112766
+ mov r4d, r4m2767
+ shl r4d, 72768
+2769
+%ifdef PIC2770
+ lea r5, [pw_ChromaCoeffVer_32_avx512]2771
+ mova m7, [r5 + r4]2772
+ mova m8, [r5 + r4 + mmsize]2773
+%else2774
+ mova m7, [pw_ChromaCoeffVer_32_avx512 + r4]2775
+ mova m8, [pw_ChromaCoeffVer_32_avx512 + r4 + mmsize]2776
+%endif2777
+2778
+%ifidn %1, sp2779
+ vbroadcasti32x4 m9, [pd_526336]2780
+ mova m10, [interp8_vsp_store_avx512]2781
+%else2782
+ add r3d, r3d2783
+%endif2784
+ add r1d, r1d2785
+ sub r0, r12786
+ lea r4, [r1 * 3]2787
+ lea r5, [r3 * 3]2788
+2789
+%rep 152790
+ PROCESS_CHROMA_VERT_S_48x4_AVX512 %12791
+ lea r0, [r0 + 4 * r1]2792
+ lea r2, [r2 + 4 * r3]2793
+%endrep2794
+ PROCESS_CHROMA_VERT_S_48x4_AVX512 %12795
+ RET2796
+%endmacro2797
+2798
+%if ARCH_X86_642799
+ FILTER_VER_S_CHROMA_48x64_AVX512 ss2800
+ FILTER_VER_S_CHROMA_48x64_AVX512 sp2801
+%endif2802
+2803
+%macro PROCESS_CHROMA_VERT_S_64x2_AVX512 12804
+ PROCESS_CHROMA_VERT_S_32x2_AVX512 %12805
+ movu m1, [r0 + mmsize]2806
+ movu m3, [r0 + r1 + mmsize]2807
+ punpcklwd m0, m1, m32808
+ pmaddwd m0, m72809
+ punpckhwd m1, m32810
+ pmaddwd m1, m72811
+ movu m4, [r0 + 2 * r1 + mmsize]2812
+ punpcklwd m2, m3, m42813
+ pmaddwd m2, m72814
+ punpckhwd m3, m42815
+ pmaddwd m3, m72816
+2817
+ movu m5, [r0 + r4 + mmsize]2818
+ punpcklwd m6, m4, m52819
+ pmaddwd m6, m82820
+ paddd m0, m62821
+ punpckhwd m4, m52822
+ pmaddwd m4, m82823
+ paddd m1, m42824
+2825
+ movu m4, [r0 + 4 * r1 + mmsize]2826
+ punpcklwd m6, m5, m42827
+ pmaddwd m6, m82828
+ paddd m2, m62829
+ punpckhwd m5, m42830
+ pmaddwd m5, m82831
+ paddd m3, m52832
+2833
+%ifidn %1, sp2834
+ paddd m0, m92835
+ paddd m1, m92836
+ paddd m2, m92837
+ paddd m3, m92838
+2839
+ psrad m0, 122840
+ psrad m1, 122841
+ psrad m2, 122842
+ psrad m3, 122843
+2844
+ packssdw m0, m12845
+ packssdw m2, m32846
+ packuswb m0, m22847
+ vpermq m0, m10, m02848
+ movu [r2 + mmsize/2], ym02849
+ vextracti32x8 [r2 + r3 + mmsize/2], m0, 12850
+%else2851
+ psrad m0, 62852
+ psrad m1, 62853
+ psrad m2, 62854
+ psrad m3, 62855
+2856
+ packssdw m0, m12857
+ packssdw m2, m32858
+ movu [r2 + mmsize], m02859
+ movu [r2 + r3 + mmsize], m22860
+%endif2861
+%endmacro2862
+2863
+%macro FILTER_VER_S_CHROMA_64xN_AVX512 22864
+INIT_ZMM avx5122865
+cglobal interp_4tap_vert_%1_64x%2, 4, 6, 112866
+ mov r4d, r4m2867
+ shl r4d, 72868
+%ifdef PIC2869
+ lea r5, [pw_ChromaCoeffVer_32_avx512]2870
+ mova m7, [r5 + r4]2871
+ mova m8, [r5 + r4 + mmsize]2872
+%else2873
+ mova m7, [pw_ChromaCoeffVer_32_avx512 + r4]2874
+ mova m8, [pw_ChromaCoeffVer_32_avx512 + r4 + mmsize]2875
+%endif2876
+2877
+%ifidn %1, sp2878
+ vbroadcasti32x4 m9, [pd_526336]2879
+ mova m10, [interp8_vsp_store_avx512]2880
+%else2881
+ add r3d, r3d2882
+%endif2883
+ add r1d, r1d2884
+ sub r0, r12885
+ lea r4, [r1 * 3]2886
+ lea r5, [r3 * 3]2887
+2888
+%rep %2/2 - 12889
+ PROCESS_CHROMA_VERT_S_64x2_AVX512 %12890
+ lea r0, [r0 + r1 * 2]2891
+ lea r2, [r2 + r3 * 2]2892
+%endrep2893
+ PROCESS_CHROMA_VERT_S_64x2_AVX512 %12894
+ RET2895
+%endmacro2896
+2897
+%if ARCH_X86_642898
+ FILTER_VER_S_CHROMA_64xN_AVX512 ss, 162899
+ FILTER_VER_S_CHROMA_64xN_AVX512 ss, 322900
+ FILTER_VER_S_CHROMA_64xN_AVX512 ss, 482901
+ FILTER_VER_S_CHROMA_64xN_AVX512 ss, 642902
+ FILTER_VER_S_CHROMA_64xN_AVX512 sp, 162903
+ FILTER_VER_S_CHROMA_64xN_AVX512 sp, 322904
+ FILTER_VER_S_CHROMA_64xN_AVX512 sp, 482905
+ FILTER_VER_S_CHROMA_64xN_AVX512 sp, 642906
+%endif2907
+;-------------------------------------------------------------------------------------------------------------2908
+;avx512 chroma_vss code end2909
+;-------------------------------------------------------------------------------------------------------------2910
+;-------------------------------------------------------------------------------------------------------------2911
+;ipfilter_chroma_avx512 code end2912
+;-------------------------------------------------------------------------------------------------------------2913
+;-------------------------------------------------------------------------------------------------------------2914
+;ipfilter_luma_avx512 code start2915
+;-------------------------------------------------------------------------------------------------------------2916
+%macro PROCESS_IPFILTER_LUMA_PP_64x1_AVX512 02917
+ ; register map2918
+ ; m0 , m1 interpolate coeff2919
+ ; m2 , m3, m4 shuffle order table2920
+ ; m5 - pw_12921
+ ; m6 - pw_5122922
+2923
+ movu m7, [r0]2924
+ movu m9, [r0 + 8]2925
+2926
+ pshufb m8, m7, m32927
+ pshufb m7, m22928
+ pshufb m10, m9, m32929
+ pshufb m11, m9, m42930
+ pshufb m9, m22931
+2932
+2933
+ pmaddubsw m7, m02934
+ pmaddubsw m12, m8, m12935
+ pmaddwd m7, m52936
+ pmaddwd m12, m52937
+ paddd m7, m122938
+2939
+ pmaddubsw m8, m02940
+ pmaddubsw m12, m9, m12941
+ pmaddwd m8, m52942
+ pmaddwd m12, m52943
+ paddd m8, m122944
+2945
+ pmaddubsw m9, m02946
+ pmaddubsw m12, m10, m12947
+ pmaddwd m9, m52948
+ pmaddwd m12, m52949
+ paddd m9, m122950
+2951
+ pmaddubsw m10, m02952
+ pmaddubsw m12, m11, m12953
+ pmaddwd m10, m52954
+ pmaddwd m12, m52955
+ paddd m10, m122956
+2957
+ packssdw m7, m82958
+ packssdw m9, m102959
+ pmulhrsw m7, m62960
+ pmulhrsw m9, m62961
+ packuswb m7, m92962
+ movu [r2], m72963
+%endmacro2964
+2965
+%macro PROCESS_IPFILTER_LUMA_PP_32x2_AVX512 02966
+ ; register map2967
+ ; m0 , m1 interpolate coeff2968
+ ; m2 , m3, m4 shuffle order table2969
+ ; m5 - pw_12970
+ ; m6 - pw_5122971
+2972
+ movu ym7, [r0]2973
+ vinserti32x8 m7, [r0 + r1], 12974
+ movu ym9, [r0 + 8]2975
+ vinserti32x8 m9, [r0 + r1 + 8], 12976
+2977
+ pshufb m8, m7, m32978
+ pshufb m7, m22979
+ pshufb m10, m9, m32980
+ pshufb m11, m9, m42981
+ pshufb m9, m22982
+2983
+ pmaddubsw m7, m02984
+ pmaddubsw m12, m8, m12985
+ pmaddwd m7, m52986
+ pmaddwd m12, m52987
+ paddd m7, m122988
+2989
+ pmaddubsw m8, m02990
+ pmaddubsw m12, m9, m12991
+ pmaddwd m8, m52992
+ pmaddwd m12, m52993
+ paddd m8, m122994
+2995
+ pmaddubsw m9, m02996
+ pmaddubsw m12, m10, m12997
+ pmaddwd m9, m52998
+ pmaddwd m12, m52999
+ paddd m9, m123000
+3001
+ pmaddubsw m10, m03002
+ pmaddubsw m12, m11, m13003
+ pmaddwd m10, m53004
+ pmaddwd m12, m53005
+ paddd m10, m123006
+3007
+ packssdw m7, m83008
+ packssdw m9, m103009
+ pmulhrsw m7, m63010
+ pmulhrsw m9, m63011
+ packuswb m7, m93012
+ movu [r2], ym73013
+ vextracti32x8 [r2 + r3], m7, 13014
+%endmacro3015
+3016
+%macro PROCESS_IPFILTER_LUMA_PP_16x4_AVX512 03017
+ ; register map3018
+ ; m0 , m1 interpolate coeff3019
+ ; m2 , m3, m4 shuffle order table3020
+ ; m5 - pw_13021
+ ; m6 - pw_5123022
+3023
+ movu xm7, [r0]3024
+ vinserti32x4 m7, [r0 + r1], 13025
+ vinserti32x4 m7, [r0 + 2 * r1], 23026
+ vinserti32x4 m7, [r0 + r6], 33027
+3028
+ pshufb m8, m7, m33029
+ pshufb m7, m23030
+3031
+ movu xm9, [r0 + 8]3032
+ vinserti32x4 m9, [r0 + r1 + 8], 13033
+ vinserti32x4 m9, [r0 + 2 * r1 + 8], 23034
+ vinserti32x4 m9, [r0 + r6 + 8], 33035
+3036
+ pshufb m10, m9, m33037
+ pshufb m11, m9, m43038
+ pshufb m9, m23039
+3040
+ pmaddubsw m7, m03041
+ pmaddubsw m12, m8, m13042
+ pmaddwd m7, m53043
+ pmaddwd m12, m53044
+ paddd m7, m123045
+3046
+ pmaddubsw m8, m03047
+ pmaddubsw m12, m9, m13048
+ pmaddwd m8, m53049
+ pmaddwd m12, m53050
+ paddd m8, m123051
+3052
+ pmaddubsw m9, m03053
+ pmaddubsw m12, m10, m13054
+ pmaddwd m9, m53055
+ pmaddwd m12, m53056
+ paddd m9, m123057
+3058
+ pmaddubsw m10, m03059
+ pmaddubsw m12, m11, m13060
+ pmaddwd m10, m53061
+ pmaddwd m12, m53062
+ paddd m10, m123063
+3064
+ packssdw m7, m83065
+ packssdw m9, m103066
+ pmulhrsw m7, m63067
+ pmulhrsw m9, m63068
+ packuswb m7, m93069
+ movu [r2], xm73070
+ vextracti32x4 [r2 + r3], m7, 13071
+ vextracti32x4 [r2 + 2 * r3], m7, 23072
+ vextracti32x4 [r2 + r7], m7, 33073
+%endmacro3074
+3075
+%macro PROCESS_IPFILTER_LUMA_PP_48x4_AVX512 03076
+ ; register map3077
+ ; m0 , m1 interpolate coeff3078
+ ; m2 , m3, m4 shuffle order table3079
+ ; m5 - pw_13080
+ ; m6 - pw_5123081
+3082
+ movu ym7, [r0]3083
+ vinserti32x8 m7, [r0 + r1], 13084
+ movu ym9, [r0 + 8]3085
+ vinserti32x8 m9, [r0 + r1 + 8], 13086
+3087
+ pshufb m8, m7, m33088
+ pshufb m7, m23089
+ pshufb m10, m9, m33090
+ pshufb m11, m9, m43091
+ pshufb m9, m23092
+3093
+ pmaddubsw m7, m03094
+ pmaddubsw m12, m8, m13095
+ pmaddwd m7, m53096
+ pmaddwd m12, m53097
+ paddd m7, m123098
+3099
+ pmaddubsw m8, m03100
+ pmaddubsw m12, m9, m13101
+ pmaddwd m8, m53102
+ pmaddwd m12, m53103
+ paddd m8, m123104
+3105
+ pmaddubsw m9, m03106
+ pmaddubsw m12, m10, m13107
+ pmaddwd m9, m53108
+ pmaddwd m12, m53109
+ paddd m9, m123110
+3111
+ pmaddubsw m10, m03112
+ pmaddubsw m12, m11, m13113
+ pmaddwd m10, m53114
+ pmaddwd m12, m53115
+ paddd m10, m123116
+3117
+ packssdw m7, m83118
+ packssdw m9, m103119
+ pmulhrsw m7, m63120
+ pmulhrsw m9, m63121
+ packuswb m7, m93122
+ movu [r2], ym73123
+ vextracti32x8 [r2 + r3], m7, 13124
+3125
+ movu ym7, [r0 + 2 * r1]3126
+ vinserti32x8 m7, [r0 + r6], 13127
+ movu ym9, [r0 + 2 * r1 + 8]3128
+ vinserti32x8 m9, [r0 + r6 + 8], 13129
+3130
+ pshufb m8, m7, m33131
+ pshufb m7, m23132
+ pshufb m10, m9, m33133
+ pshufb m11, m9, m43134
+ pshufb m9, m23135
+3136
+ pmaddubsw m7, m03137
+ pmaddubsw m12, m8, m13138
+ pmaddwd m7, m53139
+ pmaddwd m12, m53140
+ paddd m7, m123141
+3142
+ pmaddubsw m8, m03143
+ pmaddubsw m12, m9, m13144
+ pmaddwd m8, m53145
+ pmaddwd m12, m53146
+ paddd m8, m123147
+3148
+ pmaddubsw m9, m03149
+ pmaddubsw m12, m10, m13150
+ pmaddwd m9, m53151
+ pmaddwd m12, m53152
+ paddd m9, m123153
+3154
+ pmaddubsw m10, m03155
+ pmaddubsw m12, m11, m13156
+ pmaddwd m10, m53157
+ pmaddwd m12, m53158
+ paddd m10, m123159
+3160
+ packssdw m7, m83161
+ packssdw m9, m103162
+ pmulhrsw m7, m63163
+ pmulhrsw m9, m63164
+ packuswb m7, m93165
+ movu [r2 + 2 * r3], ym73166
+ vextracti32x8 [r2 + r7], m7, 13167
+3168
+ movu xm7, [r0 + mmsize/2]3169
+ vinserti32x4 m7, [r0 + r1 + mmsize/2], 13170
+ vinserti32x4 m7, [r0 + 2 * r1 + mmsize/2], 23171
+ vinserti32x4 m7, [r0 + r6 + mmsize/2], 33172
+3173
+ pshufb m8, m7, m33174
+ pshufb m7, m23175
+3176
+ movu xm9, [r0 + 40]3177
+ vinserti32x4 m9, [r0 + r1 + 40], 13178
+ vinserti32x4 m9, [r0 + 2 * r1 + 40], 23179
+ vinserti32x4 m9, [r0 + r6 + 40], 33180
+3181
+ pshufb m10, m9, m33182
+ pshufb m11, m9, m43183
+ pshufb m9, m23184
+3185
+ pmaddubsw m7, m03186
+ pmaddubsw m12, m8, m13187
+ pmaddwd m7, m53188
+ pmaddwd m12, m53189
+ paddd m7, m123190
+3191
+ pmaddubsw m8, m03192
+ pmaddubsw m12, m9, m13193
+ pmaddwd m8, m53194
+ pmaddwd m12, m53195
+ paddd m8, m123196
+3197
+ pmaddubsw m9, m03198
+ pmaddubsw m12, m10, m13199
+ pmaddwd m9, m53200
+ pmaddwd m12, m53201
+ paddd m9, m123202
+3203
+ pmaddubsw m10, m03204
+ pmaddubsw m12, m11, m13205
+ pmaddwd m10, m53206
+ pmaddwd m12, m53207
+ paddd m10, m123208
+3209
+ packssdw m7, m83210
+ packssdw m9, m103211
+ pmulhrsw m7, m63212
+ pmulhrsw m9, m63213
+ packuswb m7, m93214
+ movu [r2 + mmsize/2], xm73215
+ vextracti32x4 [r2 + r3 + mmsize/2], m7, 13216
+ vextracti32x4 [r2 + 2 * r3 + mmsize/2], m7, 23217
+ vextracti32x4 [r2 + r7 + mmsize/2], m7, 33218
+%endmacro3219
+3220
+%macro IPFILTER_LUMA_64xN_AVX512 13221
+INIT_ZMM avx5123222
+cglobal interp_8tap_horiz_pp_64x%1, 4,6,133223
+ sub r0, 33224
+ mov r4d, r4m3225
+%ifdef PIC3226
+ lea r5, [tab_LumaCoeff]3227
+ vpbroadcastd m0, [r5 + r4 * 8]3228
+ vpbroadcastd m1, [r5 + r4 * 8 + 4]3229
+%else3230
+ vpbroadcastd m0, [tab_LumaCoeff + r4 * 8]3231
+ vpbroadcastd m1, [tab_LumaCoeff + r4 * 8 + 4]3232
+%endif3233
+ vbroadcasti32x8 m2, [interp4_horiz_shuf_load1_avx512]3234
+ vbroadcasti32x8 m3, [interp4_horiz_shuf_load3_avx512]3235
+ vbroadcasti32x8 m4, [interp4_horiz_shuf_load2_avx512]3236
+ vpbroadcastd m5, [pw_1]3237
+ vbroadcasti32x8 m6, [pw_512]3238
+3239
+%rep %1-13240
+ PROCESS_IPFILTER_LUMA_PP_64x1_AVX5123241
+ lea r0, [r0 + r1]3242
+ lea r2, [r2 + r3]3243
+%endrep3244
+ PROCESS_IPFILTER_LUMA_PP_64x1_AVX5123245
+ RET3246
+%endmacro3247
+3248
+%if ARCH_X86_643249
+IPFILTER_LUMA_64xN_AVX512 163250
+IPFILTER_LUMA_64xN_AVX512 323251
+IPFILTER_LUMA_64xN_AVX512 483252
+IPFILTER_LUMA_64xN_AVX512 643253
+%endif3254
+3255
+%macro IPFILTER_LUMA_32xN_AVX512 13256
+INIT_ZMM avx5123257
+cglobal interp_8tap_horiz_pp_32x%1, 4,6,133258
+ sub r0, 33259
+ mov r4d, r4m3260
+%ifdef PIC3261
+ lea r5, [tab_LumaCoeff]3262
+ vpbroadcastd m0, [r5 + r4 * 8]3263
+ vpbroadcastd m1, [r5 + r4 * 8 + 4]3264
+%else3265
+ vpbroadcastd m0, [tab_LumaCoeff + r4 * 8]3266
+ vpbroadcastd m1, [tab_LumaCoeff + r4 * 8 + 4]3267
+%endif3268
+ vbroadcasti32x8 m2, [interp4_horiz_shuf_load1_avx512]3269
+ vbroadcasti32x8 m3, [interp4_horiz_shuf_load3_avx512]3270
+ vbroadcasti32x8 m4, [interp4_horiz_shuf_load2_avx512]3271
+ vpbroadcastd m5, [pw_1]3272
+ vbroadcasti32x8 m6, [pw_512]3273
+3274
+%rep %1/2 -13275
+ PROCESS_IPFILTER_LUMA_PP_32x2_AVX5123276
+ lea r0, [r0 + 2 * r1]3277
+ lea r2, [r2 + 2 * r3]3278
+%endrep3279
+ PROCESS_IPFILTER_LUMA_PP_32x2_AVX5123280
+ RET3281
+%endmacro3282
+3283
+%if ARCH_X86_643284
+IPFILTER_LUMA_32xN_AVX512 83285
+IPFILTER_LUMA_32xN_AVX512 163286
+IPFILTER_LUMA_32xN_AVX512 243287
+IPFILTER_LUMA_32xN_AVX512 323288
+IPFILTER_LUMA_32xN_AVX512 643289
+%endif3290
+3291
+%macro IPFILTER_LUMA_16xN_AVX512 13292
+INIT_ZMM avx5123293
+cglobal interp_8tap_horiz_pp_16x%1, 4,8,143294
+ sub r0, 33295
+ mov r4d, r4m3296
+ lea r6, [3 * r1]3297
+ lea r7, [3 * r3]3298
+%ifdef PIC3299
+ lea r5, [tab_LumaCoeff]3300
+ vpbroadcastd m0, [r5 + r4 * 8]3301
+ vpbroadcastd m1, [r5 + r4 * 8 + 4]3302
+%else3303
+ vpbroadcastd m0, [tab_LumaCoeff + r4 * 8]3304
+ vpbroadcastd m1, [tab_LumaCoeff + r4 * 8 + 4]3305
+%endif3306
+ vbroadcasti32x8 m2, [interp4_horiz_shuf_load1_avx512]3307
+ vbroadcasti32x8 m3, [interp4_horiz_shuf_load3_avx512]3308
+ vbroadcasti32x8 m4, [interp4_horiz_shuf_load2_avx512]3309
+ vpbroadcastd m5, [pw_1]3310
+ vbroadcasti32x8 m6, [pw_512]3311
+3312
+%rep %1/4 -13313
+ PROCESS_IPFILTER_LUMA_PP_16x4_AVX5123314
+ lea r0, [r0 + 4 * r1]3315
+ lea r2, [r2 + 4 * r3]3316
+%endrep3317
+ PROCESS_IPFILTER_LUMA_PP_16x4_AVX5123318
+ RET3319
+%endmacro3320
+3321
+%if ARCH_X86_643322
+IPFILTER_LUMA_16xN_AVX512 43323
+IPFILTER_LUMA_16xN_AVX512 83324
+IPFILTER_LUMA_16xN_AVX512 123325
+IPFILTER_LUMA_16xN_AVX512 163326
+IPFILTER_LUMA_16xN_AVX512 323327
+IPFILTER_LUMA_16xN_AVX512 643328
+%endif3329
+3330
+%if ARCH_X86_643331
+INIT_ZMM avx5123332
+cglobal interp_8tap_horiz_pp_48x64, 4,8,143333
+ sub r0, 33334
+ mov r4d, r4m3335
+ lea r6, [3 * r1]3336
+ lea r7, [3 * r3]3337
+%ifdef PIC3338
+ lea r5, [tab_LumaCoeff]3339
+ vpbroadcastd m0, [r5 + r4 * 8]3340
+ vpbroadcastd m1, [r5 + r4 * 8 + 4]3341
+%else3342
+ vpbroadcastd m0, [tab_LumaCoeff + r4 * 8]3343
+ vpbroadcastd m1, [tab_LumaCoeff + r4 * 8 + 4]3344
+%endif3345
+ vbroadcasti32x8 m2, [interp4_horiz_shuf_load1_avx512]3346
+ vbroadcasti32x8 m3, [interp4_horiz_shuf_load3_avx512]3347
+ vbroadcasti32x8 m4, [interp4_horiz_shuf_load2_avx512]3348
+ vpbroadcastd m5, [pw_1]3349
+ vbroadcasti32x8 m6, [pw_512]3350
+3351
+%rep 153352
+ PROCESS_IPFILTER_LUMA_PP_48x4_AVX5123353
+ lea r0, [r0 + 4 * r1]3354
+ lea r2, [r2 + 4 * r3]3355
+%endrep3356
+ PROCESS_IPFILTER_LUMA_PP_48x4_AVX5123357
+ RET3358
+%endif3359
+3360
+%macro PROCESS_IPFILTER_LUMA_PS_64x1_AVX512 03361
+ ; register map3362
+ ; m0 , m1 - interpolate coeff3363
+ ; m2 , m3, m4 - load shuffle order table3364
+ ; m5 - pw_13365
+ ; m6 - pw_20003366
+ ; m7 - store shuffle order table3367
+3368
+ movu ym8, [r0]3369
+ vinserti32x8 m8, [r0 + 8], 13370
+ pshufb m9, m8, m33371
+ pshufb m10, m8, m43372
+ pshufb m8, m23373
+3374
+ movu ym11, [r0 + mmsize/2]3375
+ vinserti32x8 m11, [r0 + mmsize/2 + 8], 13376
+ pshufb m12, m11, m33377
+ pshufb m13, m11, m43378
+ pshufb m11, m23379
+3380
+ pmaddubsw m8, m03381
+ pmaddubsw m14, m9, m13382
+ pmaddwd m8, m53383
+ pmaddwd m14, m53384
+ paddd m8, m143385
+3386
+ pmaddubsw m9, m03387
+ pmaddubsw m14, m10, m13388
+ pmaddwd m9, m53389
+ pmaddwd m14, m53390
+ paddd m9, m143391
+3392
+ pmaddubsw m11, m03393
+ pmaddubsw m14, m12, m13394
+ pmaddwd m11, m53395
+ pmaddwd m14, m53396
+ paddd m11, m143397
+3398
+ pmaddubsw m12, m03399
+ pmaddubsw m14, m13, m13400
+ pmaddwd m12, m53401
+ pmaddwd m14, m53402
+ paddd m12, m143403
+3404
+3405
+ packssdw m8, m93406
+ packssdw m11, m123407
+ psubw m8, m63408
+ psubw m11, m63409
+ vpermq m8, m7, m83410
+ vpermq m11, m7, m113411
+ movu [r2], m83412
+ movu [r2 + mmsize], m113413
+%endmacro3414
+3415
+%macro IPFILTER_LUMA_PS_64xN_AVX512 13416
+INIT_ZMM avx5123417
+cglobal interp_8tap_horiz_ps_64x%1, 4,7,153418
+ mov r4d, r4m3419
+ mov r5d, r5m3420
+3421
+%ifdef PIC3422
+ lea r6, [tab_LumaCoeff]3423
+ vpbroadcastd m0, [r6 + r4 * 8]3424
+ vpbroadcastd m1, [r6 + r4 * 8 + 4]3425
+%else3426
+ vpbroadcastd m0, [tab_LumaCoeff + r4 * 8]3427
+ vpbroadcastd m1, [tab_LumaCoeff + r4 * 8 + 4]3428
+%endif3429
+ vbroadcasti32x8 m2, [interp4_horiz_shuf_load1_avx512]3430
+ vbroadcasti32x8 m3, [interp4_horiz_shuf_load3_avx512]3431
+ vbroadcasti32x8 m4, [interp4_horiz_shuf_load2_avx512]3432
+ vpbroadcastd m5, [pw_1]3433
+ vbroadcasti32x8 m6, [pw_2000]3434
+ mova m7, [interp8_hps_store_avx512]3435
+3436
+ mov r4d, %13437
+ sub r0, 33438
+ test r5d, r5d3439
+ jz .loop3440
+ lea r6, [r1 * 3]3441
+ sub r0, r6 ; r0(src)-r63442
+ add r4d, 7 ; blkheight += N - 13443
+3444
+.loop:3445
+ PROCESS_IPFILTER_LUMA_PS_64x1_AVX5123446
+ lea r0, [r0 + r1]3447
+ lea r2, [r2 + 2 * r3]3448
+ dec r4d3449
+ jnz .loop3450
+ RET3451
+%endmacro3452
+3453
+%if ARCH_X86_64 == 13454
+ IPFILTER_LUMA_PS_64xN_AVX512 163455
+ IPFILTER_LUMA_PS_64xN_AVX512 323456
+ IPFILTER_LUMA_PS_64xN_AVX512 483457
+ IPFILTER_LUMA_PS_64xN_AVX512 643458
+%endif3459
+3460
+%macro PROCESS_IPFILTER_LUMA_PS_32x1_AVX512 03461
+ ; register map3462
+ ; m0 , m1 - interpolate coeff3463
+ ; m2 , m3, m4 - load shuffle order table3464
+ ; m5 - pw_13465
+ ; m6 - pw_20003466
+ ; m7 - store shuffle order table3467
+3468
+ movu ym8, [r0]3469
+ vinserti32x8 m8, [r0 + 8], 13470
+ pshufb m9, m8, m33471
+ pshufb m10, m8, m43472
+ pshufb m8, m23473
+3474
+ pmaddubsw m8, m03475
+ pmaddubsw m11, m9, m13476
+ pmaddwd m8, m53477
+ pmaddwd m11, m53478
+ paddd m8, m113479
+3480
+ pmaddubsw m9, m03481
+ pmaddubsw m11, m10, m13482
+ pmaddwd m9, m53483
+ pmaddwd m11, m53484
+ paddd m9, m113485
+3486
+ packssdw m8, m93487
+ psubw m8, m63488
+ vpermq m8, m7, m83489
+ movu [r2], m83490
+%endmacro3491
+3492
+%macro IPFILTER_LUMA_PS_32xN_AVX512 13493
+INIT_ZMM avx5123494
+cglobal interp_8tap_horiz_ps_32x%1, 4,7,123495
+ mov r4d, r4m3496
+ mov r5d, r5m3497
+3498
+%ifdef PIC3499
+ lea r6, [tab_LumaCoeff]3500
+ vpbroadcastd m0, [r6 + r4 * 8]3501
+ vpbroadcastd m1, [r6 + r4 * 8 + 4]3502
+%else3503
+ vpbroadcastd m0, [tab_LumaCoeff + r4 * 8]3504
+ vpbroadcastd m1, [tab_LumaCoeff + r4 * 8 + 4]3505
+%endif3506
+ vbroadcasti32x8 m2, [interp4_horiz_shuf_load1_avx512]3507
+ vbroadcasti32x8 m3, [interp4_horiz_shuf_load3_avx512]3508
+ vbroadcasti32x8 m4, [interp4_horiz_shuf_load2_avx512]3509
+ vpbroadcastd m5, [pw_1]3510
+ vbroadcasti32x8 m6, [pw_2000]3511
+ mova m7, [interp8_hps_store_avx512]3512
+3513
+ mov r4d, %13514
+ sub r0, 33515
+ test r5d, r5d3516
+ jz .loop3517
+ lea r6, [r1 * 3]3518
+ sub r0, r6 ; r0(src)-r63519
+ add r4d, 7 ; blkheight += N - 13520
+3521
+.loop:3522
+ PROCESS_IPFILTER_LUMA_PS_32x1_AVX5123523
+ lea r0, [r0 + r1]3524
+ lea r2, [r2 + 2 * r3]3525
+ dec r4d3526
+ jnz .loop3527
+ RET3528
+%endmacro3529
+3530
+%if ARCH_X86_64 == 13531
+ IPFILTER_LUMA_PS_32xN_AVX512 83532
+ IPFILTER_LUMA_PS_32xN_AVX512 163533
+ IPFILTER_LUMA_PS_32xN_AVX512 243534
+ IPFILTER_LUMA_PS_32xN_AVX512 323535
+ IPFILTER_LUMA_PS_32xN_AVX512 643536
+%endif3537
+3538
+%macro PROCESS_IPFILTER_LUMA_PS_8TAP_16x2_AVX512 03539
+ movu xm7, [r0]3540
+ vinserti32x4 m7, [r0 + 8], 13541
+ vinserti32x4 m7, [r0 + r1], 23542
+ vinserti32x4 m7, [r0 + r1 + 8], 33543
+ pshufb m8, m7, m33544
+ pshufb m9, m7, m43545
+ pshufb m7, m23546
+3547
+ pmaddubsw m7, m03548
+ pmaddubsw m10, m8, m13549
+ pmaddwd m7, m53550
+ pmaddwd m10, m53551
+ paddd m7, m103552
+3553
+ pmaddubsw m8, m03554
+ pmaddubsw m10, m9, m13555
+ pmaddwd m8, m53556
+ pmaddwd m10, m53557
+ paddd m8, m103558
+3559
+ packssdw m7, m83560
+ psubw m7, m63561
+ movu [r2], ym73562
+ vextracti32x8 [r2 + r3], m7, 13563
+%endmacro3564
+3565
+%macro PROCESS_IPFILTER_LUMA_PS_8TAP_16x1_AVX512 03566
+ movu xm7, [r0]3567
+ vinserti32x4 m7, [r0 + 8], 13568
+ pshufb ym8, ym7, ym33569
+ pshufb ym9, ym7, ym43570
+ pshufb ym7, ym23571
+3572
+ pmaddubsw ym7, ym03573
+ pmaddubsw ym10, ym8, ym13574
+ pmaddwd ym7, ym53575
+ pmaddwd ym10, ym53576
+ paddd ym7, ym103577
+3578
+ pmaddubsw ym8, ym03579
+ pmaddubsw ym10, ym9, ym13580
+ pmaddwd ym8, ym53581
+ pmaddwd ym10, ym53582
+ paddd ym8, ym103583
+3584
+ packssdw ym7, ym83585
+ psubw ym7, ym63586
+ movu [r2], ym73587
+%endmacro3588
+3589
+;-------------------------------------------------------------------------------------------------------------3590
+; void interp_horiz_ps_16xN(const pixel* src, intptr_t srcStride, int16_t* dst, intptr_t dstStride, int coeffIdx, int isRowExt)3591
+;-------------------------------------------------------------------------------------------------------------3592
+%macro IPFILTER_LUMA_PS_8TAP_16xN_AVX512 13593
+INIT_ZMM avx5123594
+cglobal interp_8tap_horiz_ps_16x%1, 4,7,113595
+ mov r4d, r4m3596
+ mov r5d, r5m3597
+ add r3, r33598
+3599
+%ifdef PIC3600
+ lea r6, [tab_LumaCoeff]3601
+ vpbroadcastd m0, [r6 + r4 * 8]3602
+ vpbroadcastd m1, [r6 + r4 * 8 + 4]3603
+%else3604
+ vpbroadcastd m0, [tab_LumaCoeff + r4 * 8]3605
+ vpbroadcastd m1, [tab_LumaCoeff + r4 * 8 + 4]3606
+%endif3607
+ vbroadcasti32x8 m2, [interp4_horiz_shuf_load1_avx512]3608
+ vbroadcasti32x8 m3, [interp4_horiz_shuf_load3_avx512]3609
+ vbroadcasti32x8 m4, [interp4_horiz_shuf_load2_avx512]3610
+ vpbroadcastd m5, [pw_1]3611
+ vbroadcasti32x8 m6, [pw_2000]3612
+3613
+ ; register map3614
+ ; m0 , m1 - interpolate coeff3615
+ ; m2 , m3, m4 - load shuffle order table3616
+ ; m5 - pw_13617
+ ; m6 - pw_20003618
+3619
+ mov r4d, %13620
+ sub r0, 33621
+ test r5d, r5d3622
+ jz .loop3623
+ lea r6, [r1 * 3]3624
+ sub r0, r6 ; r0(src)-r63625
+ add r4d, 7 ; blkheight += N - 13626
+ PROCESS_IPFILTER_LUMA_PS_8TAP_16x1_AVX5123627
+ lea r0, [r0 + r1]3628
+ lea r2, [r2 + r3]3629
+ dec r4d3630
+3631
+.loop:3632
+ PROCESS_IPFILTER_LUMA_PS_8TAP_16x2_AVX5123633
+ lea r0, [r0 + 2 * r1]3634
+ lea r2, [r2 + 2 * r3]3635
+ sub r4d, 23636
+ jnz .loop3637
+ RET3638
+%endmacro3639
+3640
+%if ARCH_X86_64 == 13641
+ IPFILTER_LUMA_PS_8TAP_16xN_AVX512 43642
+ IPFILTER_LUMA_PS_8TAP_16xN_AVX512 83643
+ IPFILTER_LUMA_PS_8TAP_16xN_AVX512 123644
+ IPFILTER_LUMA_PS_8TAP_16xN_AVX512 163645
+ IPFILTER_LUMA_PS_8TAP_16xN_AVX512 323646
+ IPFILTER_LUMA_PS_8TAP_16xN_AVX512 643647
+%endif3648
+3649
+%macro PROCESS_IPFILTER_LUMA_PS_48x1_AVX512 03650
+ ; register map3651
+ ; m0 , m1 - interpolate coeff3652
+ ; m2 , m3, m4 - load shuffle order table3653
+ ; m5 - pw_13654
+ ; m6 - pw_20003655
+ ; m7 - store shuffle order table3656
+3657
+ movu ym8, [r0]3658
+ vinserti32x8 m8, [r0 + 8], 13659
+ pshufb m9, m8, m33660
+ pshufb m10, m8, m43661
+ pshufb m8, m23662
+3663
+ pmaddubsw m8, m03664
+ pmaddubsw m11, m9, m13665
+ pmaddwd m8, m53666
+ pmaddwd m11, m53667
+ paddd m8, m113668
+3669
+ pmaddubsw m9, m03670
+ pmaddubsw m11, m10, m13671
+ pmaddwd m9, m53672
+ pmaddwd m11, m53673
+ paddd m9, m113674
+3675
+ packssdw m8, m93676
+ psubw m8, m63677
+ vpermq m8, m7, m83678
+ movu [r2], m83679
+3680
+ movu ym8, [r0 + 32]3681
+ vinserti32x4 m8, [r0 + 40], 13682
+ pshufb ym9, ym8, ym33683
+ pshufb ym10, ym8, ym43684
+ pshufb ym8, ym23685
+3686
+ pmaddubsw ym8, ym03687
+ pmaddubsw ym11, ym9, ym13688
+ pmaddwd ym8, ym53689
+ pmaddwd ym11, ym53690
+ paddd ym8, ym113691
+3692
+ pmaddubsw ym9, ym03693
+ pmaddubsw ym11, ym10, ym13694
+ pmaddwd ym9, ym53695
+ pmaddwd ym11, ym53696
+ paddd ym9, ym113697
+3698
+ packssdw ym8, ym93699
+ psubw ym8, ym63700
+ movu [r2 + mmsize], ym83701
+%endmacro3702
+3703
+;-------------------------------------------------------------------------------------------------------------3704
+; void interp_horiz_ps_48xN(const pixel* src, intptr_t srcStride, int16_t* dst, intptr_t dstStride, int coeffIdx, int isRowExt)3705
+;-------------------------------------------------------------------------------------------------------------3706
+%macro IPFILTER_LUMA_PS_48xN_AVX512 13707
+INIT_ZMM avx5123708
+cglobal interp_8tap_horiz_ps_48x%1, 4,7,123709
+ mov r4d, r4m3710
+ mov r5d, r5m3711
+3712
+%ifdef PIC3713
+ lea r6, [tab_LumaCoeff]3714
+ vpbroadcastd m0, [r6 + r4 * 8]3715
+ vpbroadcastd m1, [r6 + r4 * 8 + 4]3716
+%else3717
+ vpbroadcastd m0, [tab_LumaCoeff + r4 * 8]3718
+ vpbroadcastd m1, [tab_LumaCoeff + r4 * 8 + 4]3719
+%endif3720
+ vbroadcasti32x8 m2, [interp4_horiz_shuf_load1_avx512]3721
+ vbroadcasti32x8 m3, [interp4_horiz_shuf_load3_avx512]3722
+ vbroadcasti32x8 m4, [interp4_horiz_shuf_load2_avx512]3723
+ vpbroadcastd m5, [pw_1]3724
+ vbroadcasti32x8 m6, [pw_2000]3725
+ mova m7, [interp8_hps_store_avx512]3726
+3727
+ mov r4d, %13728
+ sub r0, 33729
+ test r5d, r5d3730
+ jz .loop3731
+ lea r6, [r1 * 3]3732
+ sub r0, r6 ; r0(src)-r63733
+ add r4d, 7 ; blkheight += N - 13734
+3735
+.loop:3736
+ PROCESS_IPFILTER_LUMA_PS_48x1_AVX5123737
+ lea r0, [r0 + r1]3738
+ lea r2, [r2 + 2 * r3]3739
+ dec r4d3740
+ jnz .loop3741
+ RET3742
+%endmacro3743
+3744
+%if ARCH_X86_64 == 13745
+ IPFILTER_LUMA_PS_48xN_AVX512 643746
+%endif3747
+3748
+;-------------------------------------------------------------------------------------------------------------3749
+;avx512 luma_vss code start3750
+;-------------------------------------------------------------------------------------------------------------3751
+%macro PROCESS_LUMA_VERT_SS_8x8_AVX512 03752
+ lea r6, [r0 + 4 * r1]3753
+ movu xm1, [r0] ;0 row3754
+ vinserti32x4 m1, [r0 + 2 * r1], 13755
+ vinserti32x4 m1, [r0 + 4 * r1], 23756
+ vinserti32x4 m1, [r6 + 2 * r1], 33757
+ movu xm3, [r0 + r1] ;1 row3758
+ vinserti32x4 m3, [r0 + r7], 13759
+ vinserti32x4 m3, [r6 + r1], 23760
+ vinserti32x4 m3, [r6 + r7], 33761
+ punpcklwd m0, m1, m33762
+ pmaddwd m0, m153763
+ punpckhwd m1, m33764
+ pmaddwd m1, m153765
+3766
+ movu xm4, [r0 + 2 * r1] ;2 row3767
+ vinserti32x4 m4, [r0 + 4 * r1], 13768
+ vinserti32x4 m4, [r6 + 2 * r1], 23769
+ vinserti32x4 m4, [r6 + 4 * r1], 33770
+ punpcklwd m2, m3, m43771
+ pmaddwd m2, m153772
+ punpckhwd m3, m43773
+ pmaddwd m3, m153774
+3775
+ lea r4, [r6 + 4 * r1]3776
+ movu xm5, [r0 + r7] ;3 row3777
+ vinserti32x4 m5, [r6 + r1], 13778
+ vinserti32x4 m5, [r6 + r7], 23779
+ vinserti32x4 m5, [r4 + r1], 33780
+ punpcklwd m6, m4, m53781
+ pmaddwd m6, m163782
+ punpckhwd m4, m53783
+ pmaddwd m4, m163784
+3785
+ paddd m0, m63786
+ paddd m1, m43787
+3788
+ movu xm4, [r0 + 4 * r1] ;4 row3789
+ vinserti32x4 m4, [r6 + 2 * r1], 13790
+ vinserti32x4 m4, [r6 + 4 * r1], 23791
+ vinserti32x4 m4, [r4 + 2 * r1], 33792
+ punpcklwd m6, m5, m43793
+ pmaddwd m6, m163794
+ punpckhwd m5, m43795
+ pmaddwd m5, m163796
+3797
+ paddd m2, m63798
+ paddd m3, m53799
+3800
+ movu xm11, [r6 + r1] ;5 row3801
+ vinserti32x4 m11, [r6 + r7], 13802
+ vinserti32x4 m11, [r4 + r1], 23803
+ vinserti32x4 m11, [r4 + r7], 33804
+ punpcklwd m8, m4, m113805
+ pmaddwd m8, m173806
+ punpckhwd m4, m113807
+ pmaddwd m4, m173808
+3809
+ movu xm12, [r6 + 2 * r1] ;6 row3810
+ vinserti32x4 m12, [r6 + 4 * r1], 13811
+ vinserti32x4 m12, [r4 + 2 * r1], 23812
+ vinserti32x4 m12, [r4 + 4 * r1], 33813
+ punpcklwd m10, m11, m123814
+ pmaddwd m10, m173815
+ punpckhwd m11, m123816
+ pmaddwd m11, m173817
+3818
+ lea r8, [r4 + 4 * r1]3819
+ movu xm13, [r6 + r7] ;7 row3820
+ vinserti32x4 m13, [r4 + r1], 13821
+ vinserti32x4 m13, [r4 + r7], 23822
+ vinserti32x4 m13, [r8 + r1], 33823
+ punpcklwd m14, m12, m133824
+ pmaddwd m14, m183825
+ punpckhwd m12, m133826
+ pmaddwd m12, m183827
+3828
+ paddd m8, m143829
+ paddd m4, m123830
+ paddd m0, m83831
+ paddd m1, m43832
+3833
+ movu xm12, [r6 + 4 * r1] ; 8 row3834
+ vinserti32x4 m12, [r4 + 2 * r1], 13835
+ vinserti32x4 m12, [r4 + 4 * r1], 23836
+ vinserti32x4 m12, [r8 + 2 * r1], 33837
+ punpcklwd m14, m13, m123838
+ pmaddwd m14, m183839
+ punpckhwd m13, m123840
+ pmaddwd m13, m183841
+3842
+ paddd m10, m143843
+ paddd m11, m133844
+ paddd m2, m103845
+ paddd m3, m113846
+3847
+ psrad m0, 63848
+ psrad m1, 63849
+ psrad m2, 63850
+ psrad m3, 63851
+3852
+ packssdw m0, m13853
+ packssdw m2, m33854
+3855
+ movu [r2], xm03856
+ movu [r2 + r3], xm23857
+ vextracti32x4 [r2 + 2 * r3], m0, 13858
+ vextracti32x4 [r2 + r5], m2, 13859
+ lea r2, [r2 + 4 * r3]3860
+ vextracti32x4 [r2], m0, 23861
+ vextracti32x4 [r2 + r3], m2, 23862
+ vextracti32x4 [r2 + 2 * r3], m0, 33863
+ vextracti32x4 [r2 + r5], m2, 33864
+%endmacro3865
+;-----------------------------------------------------------------------------------------------------------------3866
+; void interp_8tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)3867
+;-----------------------------------------------------------------------------------------------------------------3868
+%macro FILTER_VER_SS_LUMA_8xN_AVX512 13869
+INIT_ZMM avx5123870
+cglobal interp_8tap_vert_ss_8x%1, 5, 9, 193871
+ add r1d, r1d3872
+ add r3d, r3d3873
+ lea r7, [3 * r1]3874
+ sub r0, r73875
+ shl r4d, 83876
+%ifdef PIC3877
+ lea r5, [pw_LumaCoeffVer_avx512]3878
+ mova m15, [r5 + r4]3879
+ mova m16, [r5 + r4 + 1 * mmsize]3880
+ mova m17, [r5 + r4 + 2 * mmsize]3881
+ mova m18, [r5 + r4 + 3 * mmsize]3882
+%else3883
+ lea r5, [pw_LumaCoeffVer_avx512 + r4]3884
+ mova m15, [r5]3885
+ mova m16, [r5 + 1 * mmsize]3886
+ mova m17, [r5 + 2 * mmsize]3887
+ mova m18, [r5 + 3 * mmsize]3888
+%endif3889
+3890
+ lea r5, [3 * r3]3891
+%rep %1/8 - 13892
+ PROCESS_LUMA_VERT_SS_8x8_AVX5123893
+ lea r0, [r4]3894
+ lea r2, [r2 + 4 * r3]3895
+%endrep3896
+ PROCESS_LUMA_VERT_SS_8x8_AVX5123897
+ RET3898
+%endmacro3899
+3900
+%if ARCH_X86_643901
+ FILTER_VER_SS_LUMA_8xN_AVX512 83902
+ FILTER_VER_SS_LUMA_8xN_AVX512 163903
+ FILTER_VER_SS_LUMA_8xN_AVX512 323904
+%endif3905
+%macro PROCESS_LUMA_VERT_S_16x4_AVX512 13906
+ movu ym1, [r0]3907
+ movu ym3, [r0 + r1]3908
+ vinserti32x8 m1, [r0 + 2 * r1], 13909
+ vinserti32x8 m3, [r0 + r7], 13910
+ punpcklwd m0, m1, m33911
+ pmaddwd m0, m153912
+ punpckhwd m1, m33913
+ pmaddwd m1, m153914
+3915
+ lea r6, [r0 + 4 * r1]3916
+ movu ym4, [r0 + 2 * r1]3917
+ vinserti32x8 m4, [r6], 13918
+ punpcklwd m2, m3, m43919
+ pmaddwd m2, m153920
+ punpckhwd m3, m43921
+ pmaddwd m3, m153922
+3923
+ movu ym5, [r0 + r7]3924
+ vinserti32x8 m5, [r6 + r1], 13925
+ punpcklwd m6, m4, m53926
+ pmaddwd m6, m163927
+ punpckhwd m4, m53928
+ pmaddwd m4, m163929
+3930
+ paddd m0, m63931
+ paddd m1, m43932
+3933
+ movu ym4, [r6]3934
+ vinserti32x8 m4, [r6 + 2 * r1], 13935
+ punpcklwd m6, m5, m43936
+ pmaddwd m6, m163937
+ punpckhwd m5, m43938
+ pmaddwd m5, m163939
+3940
+ paddd m2, m63941
+ paddd m3, m53942
+3943
+ movu ym11, [r6 + r1]3944
+ vinserti32x8 m11, [r6 + r7], 13945
+ punpcklwd m8, m4, m113946
+ pmaddwd m8, m173947
+ punpckhwd m4, m113948
+ pmaddwd m4, m173949
+3950
+ movu ym12, [r6 + 2 * r1]3951
+ vinserti32x8 m12, [r6 + 4 * r1], 13952
+ punpcklwd m10, m11, m123953
+ pmaddwd m10, m173954
+ punpckhwd m11, m123955
+ pmaddwd m11, m173956
+3957
+ lea r4, [r6 + 4 * r1]3958
+ movu ym13, [r6 + r7]3959
+ vinserti32x8 m13, [r4 + r1], 13960
+ punpcklwd m14, m12, m133961
+ pmaddwd m14, m183962
+ punpckhwd m12, m133963
+ pmaddwd m12, m183964
+3965
+ paddd m8, m143966
+ paddd m4, m123967
+ paddd m0, m83968
+ paddd m1, m43969
+3970
+ movu ym12, [r6 + 4 * r1]3971
+ vinserti32x8 m12, [r4 + 2 * r1], 13972
+ punpcklwd m14, m13, m123973
+ pmaddwd m14, m183974
+ punpckhwd m13, m123975
+ pmaddwd m13, m183976
+3977
+ paddd m10, m143978
+ paddd m11, m133979
+ paddd m2, m103980
+ paddd m3, m113981
+%ifidn %1, sp3982
+ paddd m0, m193983
+ paddd m1, m193984
+ paddd m2, m193985
+ paddd m3, m193986
+3987
+ psrad m0, 123988
+ psrad m1, 123989
+ psrad m2, 123990
+ psrad m3, 123991
+3992
+ packssdw m0, m13993
+ packssdw m2, m33994
+ packuswb m0, m23995
+ vpermq m0, m20, m03996
+ movu [r2], xm03997
+ vextracti32x4 [r2 + r3], m0, 23998
+ vextracti32x4 [r2 + 2 * r3], m0, 13999
+ vextracti32x4 [r2 + r5], m0, 34000
+%else4001
+ psrad m0, 64002
+ psrad m1, 64003
+ psrad m2, 64004
+ psrad m3, 64005
+4006
+ packssdw m0, m14007
+ packssdw m2, m34008
+4009
+ movu [r2], ym04010
+ movu [r2 + r3], ym24011
+ vextracti32x8 [r2 + 2 * r3], m0, 14012
+ vextracti32x8 [r2 + r5], m2, 14013
+%endif4014
+%endmacro4015
+;-----------------------------------------------------------------------------------------------------------------4016
+; void interp_8tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)4017
+;-----------------------------------------------------------------------------------------------------------------4018
+%macro FILTER_VER_S_LUMA_16xN_AVX512 24019
+INIT_ZMM avx5124020
+cglobal interp_8tap_vert_%1_16x%2, 5, 8, 214021
+ add r1d, r1d4022
+ lea r7, [3 * r1]4023
+ sub r0, r74024
+ shl r4d, 84025
+%ifdef PIC4026
+ lea r5, [pw_LumaCoeffVer_avx512]4027
+ mova m15, [r5 + r4]4028
+ mova m16, [r5 + r4 + 1 * mmsize]4029
+ mova m17, [r5 + r4 + 2 * mmsize]4030
+ mova m18, [r5 + r4 + 3 * mmsize]4031
+%else4032
+ lea r5, [pw_LumaCoeffVer_avx512 + r4]4033
+ mova m15, [r5]4034
+ mova m16, [r5 + 1 * mmsize]4035
+ mova m17, [r5 + 2 * mmsize]4036
+ mova m18, [r5 + 3 * mmsize]4037
+%endif4038
+%ifidn %1, sp4039
+ vbroadcasti32x4 m19, [pd_526336]4040
+ mova m20, [interp8_vsp_store_avx512]4041
+%else4042
+ add r3d, r3d4043
+%endif4044
+4045
+ lea r5, [3 * r3]4046
+%rep %2/4 - 14047
+ PROCESS_LUMA_VERT_S_16x4_AVX512 %14048
+ lea r0, [r0 + 4 * r1]4049
+ lea r2, [r2 + 4 * r3]4050
+%endrep4051
+ PROCESS_LUMA_VERT_S_16x4_AVX512 %14052
+ RET4053
+%endmacro4054
+4055
+%if ARCH_X86_644056
+ FILTER_VER_S_LUMA_16xN_AVX512 ss, 44057
+ FILTER_VER_S_LUMA_16xN_AVX512 ss, 84058
+ FILTER_VER_S_LUMA_16xN_AVX512 ss, 124059
+ FILTER_VER_S_LUMA_16xN_AVX512 ss, 164060
+ FILTER_VER_S_LUMA_16xN_AVX512 ss, 324061
+ FILTER_VER_S_LUMA_16xN_AVX512 ss, 644062
+ FILTER_VER_S_LUMA_16xN_AVX512 sp, 44063
+ FILTER_VER_S_LUMA_16xN_AVX512 sp, 84064
+ FILTER_VER_S_LUMA_16xN_AVX512 sp, 124065
+ FILTER_VER_S_LUMA_16xN_AVX512 sp, 164066
+ FILTER_VER_S_LUMA_16xN_AVX512 sp, 324067
+ FILTER_VER_S_LUMA_16xN_AVX512 sp, 644068
+%endif4069
+%macro PROCESS_LUMA_VERT_SS_24x8_AVX512 04070
+ PROCESS_LUMA_VERT_S_16x4_AVX512 ss4071
+ lea r4, [r6 + 4 * r1]4072
+ lea r8, [r4 + 4 * r1]4073
+ movu ym1, [r6]4074
+ movu ym3, [r6 + r1]4075
+ vinserti32x8 m1, [r6 + 2 * r1], 14076
+ vinserti32x8 m3, [r6 + r7], 14077
+ punpcklwd m0, m1, m34078
+ pmaddwd m0, m154079
+ punpckhwd m1, m34080
+ pmaddwd m1, m154081
+4082
+ movu ym4, [r6 + 2 * r1]4083
+ vinserti32x8 m4, [r4], 14084
+ punpcklwd m2, m3, m44085
+ pmaddwd m2, m154086
+ punpckhwd m3, m44087
+ pmaddwd m3, m154088
+4089
+ movu ym5, [r6 + r7]4090
+ vinserti32x8 m5, [r4 + r1], 14091
+ punpcklwd m6, m4, m54092
+ pmaddwd m6, m164093
+ punpckhwd m4, m54094
+ pmaddwd m4, m164095
+4096
+ paddd m0, m64097
+ paddd m1, m44098
+4099
+ movu ym4, [r4]4100
+ vinserti32x8 m4, [r4 + 2 * r1], 14101
+ punpcklwd m6, m5, m44102
+ pmaddwd m6, m164103
+ punpckhwd m5, m44104
+ pmaddwd m5, m164105
+4106
+ paddd m2, m64107
+ paddd m3, m54108
+4109
+ movu ym11, [r4 + r1]4110
+ vinserti32x8 m11, [r4 + r7], 14111
+ punpcklwd m8, m4, m114112
+ pmaddwd m8, m174113
+ punpckhwd m4, m114114
+ pmaddwd m4, m174115
+4116
+ movu ym12, [r4 + 2 * r1]4117
+ vinserti32x8 m12, [r4 + 4 * r1], 14118
+ punpcklwd m10, m11, m124119
+ pmaddwd m10, m174120
+ punpckhwd m11, m124121
+ pmaddwd m11, m174122
+4123
+ movu ym13, [r4 + r7]4124
+ vinserti32x8 m13, [r8 + r1], 14125
+ punpcklwd m14, m12, m134126
+ pmaddwd m14, m184127
+ punpckhwd m12, m134128
+ pmaddwd m12, m184129
+4130
+ paddd m8, m144131
+ paddd m4, m124132
+ paddd m0, m84133
+ paddd m1, m44134
+4135
+ movu ym12, [r4 + 4 * r1]4136
+ vinserti32x8 m12, [r8 + 2 * r1], 14137
+ punpcklwd m14, m13, m124138
+ pmaddwd m14, m184139
+ punpckhwd m13, m124140
+ pmaddwd m13, m184141
+4142
+ paddd m10, m144143
+ paddd m11, m134144
+ paddd m2, m104145
+ paddd m3, m114146
+4147
+ psrad m0, 64148
+ psrad m1, 64149
+ psrad m2, 64150
+ psrad m3, 64151
+4152
+ packssdw m0, m14153
+ packssdw m2, m34154
+4155
+ lea r9, [r2 + 4 * r3]4156
+ movu [r9], ym04157
+ movu [r9 + r3], ym24158
+ vextracti32x8 [r9 + 2 * r3], m0, 14159
+ vextracti32x8 [r9 + r5], m2, 14160
+4161
+ movu xm1, [r0 + mmsize/2]4162
+ vinserti32x4 m1, [r0 + 2 * r1 + mmsize/2], 14163
+ vinserti32x4 m1, [r0 + 4 * r1 + mmsize/2], 24164
+ vinserti32x4 m1, [r6 + 2 * r1 + mmsize/2], 34165
+ movu xm3, [r0 + r1 + mmsize/2]4166
+ vinserti32x4 m3, [r0 + r7 + mmsize/2], 14167
+ vinserti32x4 m3, [r6 + r1 + mmsize/2], 24168
+ vinserti32x4 m3, [r6 + r7 + mmsize/2], 34169
+ punpcklwd m0, m1, m34170
+ pmaddwd m0, m154171
+ punpckhwd m1, m34172
+ pmaddwd m1, m154173
+4174
+ movu xm4, [r0 + 2 * r1 + mmsize/2]4175
+ vinserti32x4 m4, [r0 + 4 * r1 + mmsize/2], 14176
+ vinserti32x4 m4, [r6 + 2 * r1 + mmsize/2], 24177
+ vinserti32x4 m4, [r6 + 4 * r1 + mmsize/2], 34178
+ punpcklwd m2, m3, m44179
+ pmaddwd m2, m154180
+ punpckhwd m3, m44181
+ pmaddwd m3, m154182
+4183
+ movu xm5, [r0 + r7 + mmsize/2]4184
+ vinserti32x4 m5, [r6 + r1 + mmsize/2], 14185
+ vinserti32x4 m5, [r6 + r7 + mmsize/2], 24186
+ vinserti32x4 m5, [r4 + r1 + mmsize/2], 34187
+ punpcklwd m6, m4, m54188
+ pmaddwd m6, m164189
+ punpckhwd m4, m54190
+ pmaddwd m4, m164191
+4192
+ paddd m0, m64193
+ paddd m1, m44194
+4195
+ movu xm4, [r0 + 4 * r1 + mmsize/2]4196
+ vinserti32x4 m4, [r6 + 2 * r1 + mmsize/2], 14197
+ vinserti32x4 m4, [r6 + 4 * r1 + mmsize/2], 24198
+ vinserti32x4 m4, [r4 + 2 * r1 + mmsize/2], 34199
+ punpcklwd m6, m5, m44200
+ pmaddwd m6, m164201
+ punpckhwd m5, m44202
+ pmaddwd m5, m164203
+4204
+ paddd m2, m64205
+ paddd m3, m54206
+4207
+ movu xm11, [r6 + r1 + mmsize/2]4208
+ vinserti32x4 m11, [r6 + r7 + mmsize/2], 14209
+ vinserti32x4 m11, [r4 + r1 + mmsize/2], 24210
+ vinserti32x4 m11, [r4 + r7 + mmsize/2], 34211
+ punpcklwd m8, m4, m114212
+ pmaddwd m8, m174213
+ punpckhwd m4, m114214
+ pmaddwd m4, m174215
+4216
+ movu xm12, [r6 + 2 * r1 + mmsize/2]4217
+ vinserti32x4 m12, [r6 + 4 * r1 + mmsize/2], 14218
+ vinserti32x4 m12, [r4 + 2 * r1 + mmsize/2], 24219
+ vinserti32x4 m12, [r4 + 4 * r1 + mmsize/2], 34220
+ punpcklwd m10, m11, m124221
+ pmaddwd m10, m174222
+ punpckhwd m11, m124223
+ pmaddwd m11, m174224
+4225
+ movu xm13, [r6 + r7 + mmsize/2]4226
+ vinserti32x4 m13, [r4 + r1 + mmsize/2], 14227
+ vinserti32x4 m13, [r4 + r7 + mmsize/2], 24228
+ vinserti32x4 m13, [r8 + r1 + mmsize/2], 34229
+ punpcklwd m14, m12, m134230
+ pmaddwd m14, m184231
+ punpckhwd m12, m134232
+ pmaddwd m12, m184233
+4234
+ paddd m8, m144235
+ paddd m4, m124236
+ paddd m0, m84237
+ paddd m1, m44238
+4239
+ movu xm12, [r6 + 4 * r1 + mmsize/2]4240
+ vinserti32x4 m12, [r4 + 2 * r1 + mmsize/2], 14241
+ vinserti32x4 m12, [r4 + 4 * r1 + mmsize/2], 24242
+ vinserti32x4 m12, [r8 + 2 * r1 + mmsize/2], 34243
+ punpcklwd m14, m13, m124244
+ pmaddwd m14, m184245
+ punpckhwd m13, m124246
+ pmaddwd m13, m184247
+4248
+ paddd m10, m144249
+ paddd m11, m134250
+ paddd m2, m104251
+ paddd m3, m114252
+4253
+ psrad m0, 64254
+ psrad m1, 64255
+ psrad m2, 64256
+ psrad m3, 64257
+4258
+ packssdw m0, m14259
+ packssdw m2, m34260
+4261
+ movu [r2 + mmsize/2], xm04262
+ movu [r2 + r3 + mmsize/2], xm24263
+ vextracti32x4 [r2 + 2 * r3 + mmsize/2], m0, 14264
+ vextracti32x4 [r2 + r5 + mmsize/2], m2, 14265
+ lea r2, [r2 + 4 * r3]4266
+ vextracti32x4 [r2 + mmsize/2], m0, 24267
+ vextracti32x4 [r2 + r3 + mmsize/2], m2, 24268
+ vextracti32x4 [r2 + 2 * r3 + mmsize/2], m0, 34269
+ vextracti32x4 [r2 + r5 + mmsize/2], m2, 34270
+%endmacro4271
+;-----------------------------------------------------------------------------------------------------------------4272
+; void interp_8tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)4273
+;-----------------------------------------------------------------------------------------------------------------4274
+%if ARCH_X86_644275
+INIT_ZMM avx5124276
+cglobal interp_8tap_vert_ss_24x32, 5, 10, 194277
+ add r1d, r1d4278
+ add r3d, r3d4279
+ lea r7, [3 * r1]4280
+ sub r0, r74281
+ shl r4d, 84282
+%ifdef PIC4283
+ lea r5, [pw_LumaCoeffVer_avx512]4284
+ mova m15, [r5 + r4]4285
+ mova m16, [r5 + r4 + 1 * mmsize]4286
+ mova m17, [r5 + r4 + 2 * mmsize]4287
+ mova m18, [r5 + r4 + 3 * mmsize]4288
+%else4289
+ lea r5, [pw_LumaCoeffVer_avx512 + r4]4290
+ mova m15, [r5]4291
+ mova m16, [r5 + 1 * mmsize]4292
+ mova m17, [r5 + 2 * mmsize]4293
+ mova m18, [r5 + 3 * mmsize]4294
+%endif4295
+4296
+ lea r5, [3 * r3]4297
+%rep 34298
+ PROCESS_LUMA_VERT_SS_24x8_AVX5124299
+ lea r0, [r4]4300
+ lea r2, [r2 + 4 * r3]4301
+%endrep4302
+ PROCESS_LUMA_VERT_SS_24x8_AVX5124303
+ RET4304
+%endif4305
+4306
+%macro PROCESS_LUMA_VERT_S_32x2_AVX512 14307
+ movu m1, [r0] ;0 row4308
+ movu m3, [r0 + r1] ;1 row4309
+ punpcklwd m0, m1, m34310
+ pmaddwd m0, m154311
+ punpckhwd m1, m34312
+ pmaddwd m1, m154313
+4314
+ movu m4, [r0 + 2 * r1] ;2 row4315
+ punpcklwd m2, m3, m44316
+ pmaddwd m2, m154317
+ punpckhwd m3, m44318
+ pmaddwd m3, m154319
+4320
+ movu m5, [r0 + r7] ;3 row4321
+ punpcklwd m6, m4, m54322
+ pmaddwd m6, m164323
+ punpckhwd m4, m54324
+ pmaddwd m4, m164325
+4326
+ paddd m0, m64327
+ paddd m1, m44328
+4329
+ movu m4, [r0 + 4 * r1] ;4 row4330
+ punpcklwd m6, m5, m44331
+ pmaddwd m6, m164332
+ punpckhwd m5, m44333
+ pmaddwd m5, m164334
+4335
+ paddd m2, m64336
+ paddd m3, m54337
+4338
+ lea r6, [r0 + 4 * r1]4339
+4340
+ movu m11, [r6 + r1] ;5 row4341
+ punpcklwd m8, m4, m114342
+ pmaddwd m8, m174343
+ punpckhwd m4, m114344
+ pmaddwd m4, m174345
+4346
+ movu m12, [r6 + 2 * r1] ;6 row4347
+ punpcklwd m10, m11, m124348
+ pmaddwd m10, m174349
+ punpckhwd m11, m124350
+ pmaddwd m11, m174351
+4352
+ movu m13, [r6 + r7] ;7 row4353
+ punpcklwd m14, m12, m134354
+ pmaddwd m14, m184355
+ punpckhwd m12, m134356
+ pmaddwd m12, m184357
+ paddd m8, m144358
+ paddd m4, m124359
+ movu m12, [r6 + 4 * r1] ; 8 row4360
+ punpcklwd m14, m13, m124361
+ pmaddwd m14, m184362
+ punpckhwd m13, m124363
+ pmaddwd m13, m184364
+ paddd m10, m144365
+ paddd m11, m134366
+4367
+ paddd m0, m84368
+ paddd m1, m44369
+ paddd m2, m104370
+ paddd m3, m114371
+%ifidn %1, sp4372
+ paddd m0, m194373
+ paddd m1, m194374
+ paddd m2, m194375
+ paddd m3, m194376
+4377
+ psrad m0, 124378
+ psrad m1, 124379
+ psrad m2, 124380
+ psrad m3, 124381
+4382
+ packssdw m0, m14383
+ packssdw m2, m34384
+ packuswb m0, m24385
+ vpermq m0, m20, m04386
+ movu [r2], ym04387
+ vextracti32x8 [r2 + r3], m0, 14388
+%else4389
+ psrad m0, 64390
+ psrad m1, 64391
+ psrad m2, 64392
+ psrad m3, 64393
+4394
+ packssdw m0, m14395
+ packssdw m2, m34396
+ movu [r2], m04397
+ movu [r2 + r3], m24398
+%endif4399
+%endmacro4400
+;-----------------------------------------------------------------------------------------------------------------4401
+; void interp_8tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)4402
+;-----------------------------------------------------------------------------------------------------------------4403
+%macro FILTER_VER_S_LUMA_32xN_AVX512 24404
+INIT_ZMM avx5124405
+cglobal interp_8tap_vert_%1_32x%2, 5, 8, 214406
+ add r1d, r1d4407
+ lea r7, [3 * r1]4408
+ sub r0, r74409
+ shl r4d, 84410
+%ifdef PIC4411
+ lea r5, [pw_LumaCoeffVer_avx512]4412
+ mova m15, [r5 + r4]4413
+ mova m16, [r5 + r4 + 1 * mmsize]4414
+ mova m17, [r5 + r4 + 2 * mmsize]4415
+ mova m18, [r5 + r4 + 3 * mmsize]4416
+%else4417
+ lea r5, [pw_LumaCoeffVer_avx512 + r4]4418
+ mova m15, [r5]4419
+ mova m16, [r5 + 1 * mmsize]4420
+ mova m17, [r5 + 2 * mmsize]4421
+ mova m18, [r5 + 3 * mmsize]4422
+%endif4423
+%ifidn %1, sp4424
+ vbroadcasti32x4 m19, [pd_526336]4425
+ mova m20, [interp8_vsp_store_avx512]4426
+%else4427
+ add r3d, r3d4428
+%endif4429
+4430
+%rep %2/2 - 14431
+ PROCESS_LUMA_VERT_S_32x2_AVX512 %14432
+ lea r0, [r0 + 2 * r1]4433
+ lea r2, [r2 + 2 * r3]4434
+%endrep4435
+ PROCESS_LUMA_VERT_S_32x2_AVX512 %14436
+ RET4437
+%endmacro4438
+4439
+%if ARCH_X86_644440
+ FILTER_VER_S_LUMA_32xN_AVX512 ss, 84441
+ FILTER_VER_S_LUMA_32xN_AVX512 ss, 164442
+ FILTER_VER_S_LUMA_32xN_AVX512 ss, 324443
+ FILTER_VER_S_LUMA_32xN_AVX512 ss, 244444
+ FILTER_VER_S_LUMA_32xN_AVX512 ss, 644445
+ FILTER_VER_S_LUMA_32xN_AVX512 sp, 84446
+ FILTER_VER_S_LUMA_32xN_AVX512 sp, 164447
+ FILTER_VER_S_LUMA_32xN_AVX512 sp, 324448
+ FILTER_VER_S_LUMA_32xN_AVX512 sp, 244449
+ FILTER_VER_S_LUMA_32xN_AVX512 sp, 644450
+%endif4451
+4452
+%macro PROCESS_LUMA_VERT_S_48x4_AVX512 14453
+ PROCESS_LUMA_VERT_S_32x2_AVX512 %14454
+ movu m1, [r0 + 2 * r1]4455
+ movu m3, [r0 + r7]4456
+ punpcklwd m0, m1, m34457
+ pmaddwd m0, m154458
+ punpckhwd m1, m34459
+ pmaddwd m1, m154460
+4461
+ movu m4, [r0 + 4 * r1]4462
+ punpcklwd m2, m3, m44463
+ pmaddwd m2, m154464
+ punpckhwd m3, m44465
+ pmaddwd m3, m154466
+4467
+ movu m5, [r6 + r1]4468
+ punpcklwd m6, m4, m54469
+ pmaddwd m6, m164470
+ punpckhwd m4, m54471
+ pmaddwd m4, m164472
+4473
+ paddd m0, m64474
+ paddd m1, m44475
+4476
+ lea r4, [r6 + 4 * r1]4477
+4478
+ movu m4, [r6 + 2 * r1]4479
+ punpcklwd m6, m5, m44480
+ pmaddwd m6, m164481
+ punpckhwd m5, m44482
+ pmaddwd m5, m164483
+4484
+ paddd m2, m64485
+ paddd m3, m54486
+4487
+ movu m11, [r6 + r7]4488
+ punpcklwd m8, m4, m114489
+ pmaddwd m8, m174490
+ punpckhwd m4, m114491
+ pmaddwd m4, m174492
+4493
+ movu m12, [r4]4494
+ punpcklwd m10, m11, m124495
+ pmaddwd m10, m174496
+ punpckhwd m11, m124497
+ pmaddwd m11, m174498
+4499
+ movu m13, [r4 + r1]4500
+ punpcklwd m14, m12, m134501
+ pmaddwd m14, m184502
+ punpckhwd m12, m134503
+ pmaddwd m12, m184504
+ paddd m8, m144505
+ paddd m4, m124506
+ movu m12, [r4 + 2 * r1]4507
+ punpcklwd m14, m13, m124508
+ pmaddwd m14, m184509
+ punpckhwd m13, m124510
+ pmaddwd m13, m184511
+ paddd m10, m144512
+ paddd m11, m134513
+4514
+ paddd m0, m84515
+ paddd m1, m44516
+ paddd m2, m104517
+ paddd m3, m114518
+%ifidn %1, sp4519
+ paddd m0, m194520
+ paddd m1, m194521
+ paddd m2, m194522
+ paddd m3, m194523
+4524
+ psrad m0, 124525
+ psrad m1, 124526
+ psrad m2, 124527
+ psrad m3, 124528
+4529
+ packssdw m0, m14530
+ packssdw m2, m34531
+ packuswb m0, m24532
+ vpermq m0, m20, m04533
+ movu [r2 + 2 * r3], ym04534
+ vextracti32x8 [r2 + r5], m0, 14535
+%else4536
+ psrad m0, 64537
+ psrad m1, 64538
+ psrad m2, 64539
+ psrad m3, 64540
+4541
+ packssdw m0, m14542
+ packssdw m2, m34543
+ movu [r2 + 2 * r3], m04544
+ movu [r2 + r5], m24545
+%endif4546
+ movu ym1, [r0 + mmsize]4547
+ movu ym3, [r0 + r1 + mmsize]4548
+ vinserti32x8 m1, [r0 + 2 * r1 + mmsize], 14549
+ vinserti32x8 m3, [r0 + r7 + mmsize], 14550
+ punpcklwd m0, m1, m34551
+ pmaddwd m0, m154552
+ punpckhwd m1, m34553
+ pmaddwd m1, m154554
+4555
+ movu ym4, [r0 + 2 * r1 + mmsize]4556
+ vinserti32x8 m4, [r6 + mmsize], 14557
+ punpcklwd m2, m3, m44558
+ pmaddwd m2, m154559
+ punpckhwd m3, m44560
+ pmaddwd m3, m154561
+4562
+ movu ym5, [r0 + r7 + mmsize]4563
+ vinserti32x8 m5, [r6 + r1 + mmsize], 14564
+ punpcklwd m6, m4, m54565
+ pmaddwd m6, m164566
+ punpckhwd m4, m54567
+ pmaddwd m4, m164568
+4569
+ paddd m0, m64570
+ paddd m1, m44571
+4572
+ movu ym4, [r6 + mmsize]4573
+ vinserti32x8 m4, [r6 + 2 * r1 + mmsize], 14574
+ punpcklwd m6, m5, m44575
+ pmaddwd m6, m164576
+ punpckhwd m5, m44577
+ pmaddwd m5, m164578
+4579
+ paddd m2, m64580
+ paddd m3, m54581
+4582
+ movu ym11, [r6 + r1 + mmsize]4583
+ vinserti32x8 m11, [r6 + r7 + mmsize], 14584
+ punpcklwd m8, m4, m114585
+ pmaddwd m8, m174586
+ punpckhwd m4, m114587
+ pmaddwd m4, m174588
+4589
+ movu ym12, [r6 + 2 * r1 + mmsize]4590
+ vinserti32x8 m12, [r6 + 4 * r1 + mmsize], 14591
+ punpcklwd m10, m11, m124592
+ pmaddwd m10, m174593
+ punpckhwd m11, m124594
+ pmaddwd m11, m174595
+4596
+ movu ym13, [r6 + r7 + mmsize]4597
+ vinserti32x8 m13, [r4 + r1 + mmsize], 14598
+ punpcklwd m14, m12, m134599
+ pmaddwd m14, m184600
+ punpckhwd m12, m134601
+ pmaddwd m12, m184602
+ paddd m8, m144603
+ paddd m4, m124604
+ movu ym12, [r6 + 4 * r1 + mmsize]4605
+ vinserti32x8 m12, [r4 + 2 * r1 + mmsize], 14606
+ punpcklwd m14, m13, m124607
+ pmaddwd m14, m184608
+ punpckhwd m13, m124609
+ pmaddwd m13, m184610
+ paddd m10, m144611
+ paddd m11, m134612
+4613
+ paddd m0, m84614
+ paddd m1, m44615
+ paddd m2, m104616
+ paddd m3, m114617
+%ifidn %1, sp4618
+ paddd m0, m194619
+ paddd m1, m194620
+ paddd m2, m194621
+ paddd m3, m194622
+4623
+ psrad m0, 124624
+ psrad m1, 124625
+ psrad m2, 124626
+ psrad m3, 124627
+4628
+ packssdw m0, m14629
+ packssdw m2, m34630
+ packuswb m0, m24631
+ vpermq m0, m20, m04632
+ movu [r2 + mmsize/2], xm04633
+ vextracti32x4 [r2 + r3 + mmsize/2], m0, 24634
+ vextracti32x4 [r2 + 2 * r3 + mmsize/2], m0, 14635
+ vextracti32x4 [r2 + r5 + mmsize/2], m0, 34636
+%else4637
+ psrad m0, 64638
+ psrad m1, 64639
+ psrad m2, 64640
+ psrad m3, 64641
+4642
+ packssdw m0, m14643
+ packssdw m2, m34644
+4645
+ movu [r2 + mmsize], ym04646
+ movu [r2 + r3 + mmsize], ym24647
+ vextracti32x8 [r2 + 2 * r3 + mmsize], m0, 14648
+ vextracti32x8 [r2 + r5 + mmsize], m2, 14649
+%endif4650
+%endmacro4651
+;-----------------------------------------------------------------------------------------------------------------4652
+; void interp_4tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)4653
+;-----------------------------------------------------------------------------------------------------------------4654
+%macro FILTER_VER_S_LUMA_48x64_AVX512 14655
+INIT_ZMM avx5124656
+cglobal interp_8tap_vert_%1_48x64, 5, 8, 214657
+ add r1d, r1d4658
+ lea r7, [3 * r1]4659
+ sub r0, r74660
+ shl r4d, 84661
+%ifdef PIC4662
+ lea r5, [pw_LumaCoeffVer_avx512]4663
+ mova m15, [r5 + r4]4664
+ mova m16, [r5 + r4 + 1 * mmsize]4665
+ mova m17, [r5 + r4 + 2 * mmsize]4666
+ mova m18, [r5 + r4 + 3 * mmsize]4667
+%else4668
+ lea r5, [pw_LumaCoeffVer_avx512 + r4]4669
+ mova m15, [r5]4670
+ mova m16, [r5 + 1 * mmsize]4671
+ mova m17, [r5 + 2 * mmsize]4672
+ mova m18, [r5 + 3 * mmsize]4673
+%endif4674
+%ifidn %1, sp4675
+ vbroadcasti32x4 m19, [pd_526336]4676
+ mova m20, [interp8_vsp_store_avx512]4677
+%else4678
+ add r3d, r3d4679
+%endif4680
+4681
+ lea r5, [3 * r3]4682
+%rep 154683
+ PROCESS_LUMA_VERT_S_48x4_AVX512 %14684
+ lea r0, [r0 + 4 * r1]4685
+ lea r2, [r2 + 4 * r3]4686
+%endrep4687
+ PROCESS_LUMA_VERT_S_48x4_AVX512 %14688
+ RET4689
+%endmacro4690
+4691
+%if ARCH_X86_644692
+ FILTER_VER_S_LUMA_48x64_AVX512 ss4693
+ FILTER_VER_S_LUMA_48x64_AVX512 sp4694
+%endif4695
+4696
+%macro PROCESS_LUMA_VERT_S_64x2_AVX512 14697
+ PROCESS_LUMA_VERT_S_32x2_AVX512 %14698
+ movu m1, [r0 + mmsize] ;0 row4699
+ movu m3, [r0 + r1 + mmsize] ;1 row4700
+ punpcklwd m0, m1, m34701
+ pmaddwd m0, m154702
+ punpckhwd m1, m34703
+ pmaddwd m1, m154704
+4705
+ movu m4, [r0 + 2 * r1 + mmsize] ;2 row4706
+ punpcklwd m2, m3, m44707
+ pmaddwd m2, m154708
+ punpckhwd m3, m44709
+ pmaddwd m3, m154710
+4711
+ movu m5, [r0 + r7 + mmsize] ;3 row4712
+ punpcklwd m6, m4, m54713
+ pmaddwd m6, m164714
+ punpckhwd m4, m54715
+ pmaddwd m4, m164716
+4717
+ paddd m0, m64718
+ paddd m1, m44719
+4720
+ movu m4, [r0 + 4 * r1 + mmsize] ;4 row4721
+ punpcklwd m6, m5, m44722
+ pmaddwd m6, m164723
+ punpckhwd m5, m44724
+ pmaddwd m5, m164725
+4726
+ paddd m2, m64727
+ paddd m3, m54728
+4729
+ movu m11, [r6 + r1 + mmsize] ;5 row4730
+ punpcklwd m8, m4, m114731
+ pmaddwd m8, m174732
+ punpckhwd m4, m114733
+ pmaddwd m4, m174734
+4735
+ movu m12, [r6 + 2 * r1 + mmsize] ;6 row4736
+ punpcklwd m10, m11, m124737
+ pmaddwd m10, m174738
+ punpckhwd m11, m124739
+ pmaddwd m11, m174740
+4741
+ movu m13, [r6 + r7 + mmsize] ;7 row4742
+ punpcklwd m14, m12, m134743
+ pmaddwd m14, m184744
+ punpckhwd m12, m134745
+ pmaddwd m12, m184746
+ paddd m8, m144747
+ paddd m4, m124748
+ movu m12, [r6 + 4 * r1 + mmsize] ; 8 row4749
+ punpcklwd m14, m13, m124750
+ pmaddwd m14, m184751
+ punpckhwd m13, m124752
+ pmaddwd m13, m184753
+ paddd m10, m144754
+ paddd m11, m134755
+4756
+ paddd m0, m84757
+ paddd m1, m44758
+ paddd m2, m104759
+ paddd m3, m114760
+%ifidn %1, sp4761
+ paddd m0, m194762
+ paddd m1, m194763
+ paddd m2, m194764
+ paddd m3, m194765
+4766
+ psrad m0, 124767
+ psrad m1, 124768
+ psrad m2, 124769
+ psrad m3, 124770
+4771
+ packssdw m0, m14772
+ packssdw m2, m34773
+ packuswb m0, m24774
+ vpermq m0, m20, m04775
+ movu [r2 + mmsize/2], ym04776
+ vextracti32x8 [r2 + r3 + mmsize/2], m0, 14777
+%else4778
+ psrad m0, 64779
+ psrad m1, 64780
+ psrad m2, 64781
+ psrad m3, 64782
+ packssdw m0, m14783
+ packssdw m2, m34784
+ movu [r2 + mmsize], m04785
+ movu [r2 + r3 + mmsize], m24786
+%endif4787
+%endmacro4788
+;-----------------------------------------------------------------------------------------------------------------4789
+; void interp_8tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)4790
+;-----------------------------------------------------------------------------------------------------------------4791
+%macro FILTER_VER_S_LUMA_64xN_AVX512 24792
+INIT_ZMM avx5124793
+cglobal interp_8tap_vert_%1_64x%2, 5, 8, 214794
+ add r1d, r1d4795
+ lea r7, [3 * r1]4796
+ sub r0, r74797
+ shl r4d, 84798
+%ifdef PIC4799
+ lea r5, [pw_LumaCoeffVer_avx512]4800
+ mova m15, [r5 + r4]4801
+ mova m16, [r5 + r4 + 1 * mmsize]4802
+ mova m17, [r5 + r4 + 2 * mmsize]4803
+ mova m18, [r5 + r4 + 3 * mmsize]4804
+%else4805
+ lea r5, [pw_LumaCoeffVer_avx512 + r4]4806
+ mova m15, [r5]4807
+ mova m16, [r5 + 1 * mmsize]4808
+ mova m17, [r5 + 2 * mmsize]4809
+ mova m18, [r5 + 3 * mmsize]4810
+%endif4811
+%ifidn %1, sp4812
+ vbroadcasti32x4 m19, [pd_526336]4813
+ mova m20, [interp8_vsp_store_avx512]4814
+%else4815
+ add r3d, r3d4816
+%endif4817
+4818
+%rep %2/2 - 14819
+ PROCESS_LUMA_VERT_S_64x2_AVX512 %14820
+ lea r0, [r0 + 2 * r1]4821
+ lea r2, [r2 + 2 * r3]4822
+%endrep4823
+ PROCESS_LUMA_VERT_S_64x2_AVX512 %14824
+ RET4825
+%endmacro4826
+4827
+%if ARCH_X86_644828
+ FILTER_VER_S_LUMA_64xN_AVX512 ss, 164829
+ FILTER_VER_S_LUMA_64xN_AVX512 ss, 324830
+ FILTER_VER_S_LUMA_64xN_AVX512 ss, 484831
+ FILTER_VER_S_LUMA_64xN_AVX512 ss, 644832
+ FILTER_VER_S_LUMA_64xN_AVX512 sp, 164833
+ FILTER_VER_S_LUMA_64xN_AVX512 sp, 324834
+ FILTER_VER_S_LUMA_64xN_AVX512 sp, 484835
+ FILTER_VER_S_LUMA_64xN_AVX512 sp, 644836
+%endif4837
+;-------------------------------------------------------------------------------------------------------------4838
+;avx512 luma_vss code end4839
+;-------------------------------------------------------------------------------------------------------------4840
+;-------------------------------------------------------------------------------------------------------------4841
+;avx512 luma_vpp and luma_vps code start4842
+;-------------------------------------------------------------------------------------------------------------4843
+%macro PROCESS_LUMA_VERT_16x8_AVX512 14844
+ lea r5, [r0 + 4 * r1]4845
+ lea r4, [r5 + 4 * r1]4846
+ movu xm1, [r0]4847
+ vinserti32x4 m1, [r0 + 2 * r1], 14848
+ vinserti32x4 m1, [r5], 24849
+ vinserti32x4 m1, [r5 + 2 * r1], 34850
+ movu xm3, [r0 + r1]4851
+ vinserti32x4 m3, [r0 + r6], 14852
+ vinserti32x4 m3, [r5 + r1], 24853
+ vinserti32x4 m3, [r5 + r6], 34854
+ punpcklbw m0, m1, m34855
+ pmaddubsw m0, m84856
+ punpckhbw m1, m34857
+ pmaddubsw m1, m84858
+4859
+ movu xm4, [r0 + 2 * r1]4860
+ vinserti32x4 m4, [r0 + 4 * r1], 14861
+ vinserti32x4 m4, [r5 + 2 * r1], 24862
+ vinserti32x4 m4, [r5 + 4 * r1], 34863
+ punpcklbw m2, m3, m44864
+ pmaddubsw m2, m84865
+ punpckhbw m3, m44866
+ pmaddubsw m3, m84867
+4868
+ movu xm5, [r0 + r6]4869
+ vinserti32x4 m5, [r5 + r1], 14870
+ vinserti32x4 m5, [r5 + r6], 24871
+ vinserti32x4 m5, [r4 + r1], 34872
+ punpcklbw m6, m4, m54873
+ pmaddubsw m6, m94874
+ punpckhbw m4, m54875
+ pmaddubsw m4, m94876
+4877
+ paddw m0, m64878
+ paddw m1, m44879
+4880
+ movu xm4, [r0 + 4 * r1]4881
+ vinserti32x4 m4, [r5 + 2 * r1], 14882
+ vinserti32x4 m4, [r5 + 4 * r1], 24883
+ vinserti32x4 m4, [r4 + 2 * r1], 34884
+ punpcklbw m6, m5, m44885
+ pmaddubsw m6, m94886
+ punpckhbw m5, m44887
+ pmaddubsw m5, m94888
+4889
+ paddw m2, m64890
+ paddw m3, m54891
+4892
+ movu xm15, [r5 + r1]4893
+ vinserti32x4 m15, [r5 + r6], 14894
+ vinserti32x4 m15, [r4 + r1], 24895
+ vinserti32x4 m15, [r4 + r6], 34896
+ punpcklbw m12, m4, m154897
+ pmaddubsw m12, m104898
+ punpckhbw m13, m4, m154899
+ pmaddubsw m13, m104900
+4901
+ lea r8, [r4 + 4 * r1]4902
+ movu xm4, [r5 + 2 * r1]4903
+ vinserti32x4 m4, [r5 + 4 * r1], 14904
+ vinserti32x4 m4, [r4 + 2 * r1], 24905
+ vinserti32x4 m4, [r4 + 4 * r1], 34906
+ punpcklbw m14, m15, m44907
+ pmaddubsw m14, m104908
+ punpckhbw m15, m44909
+ pmaddubsw m15, m104910
+4911
+ movu xm5, [r5 + r6]4912
+ vinserti32x4 m5, [r4 + r1], 14913
+ vinserti32x4 m5, [r4 + r6], 24914
+ vinserti32x4 m5, [r8 + r1], 34915
+ punpcklbw m6, m4, m54916
+ pmaddubsw m6, m114917
+ punpckhbw m4, m54918
+ pmaddubsw m4, m114919
+4920
+ paddw m12, m64921
+ paddw m13, m44922
+4923
+ movu xm4, [r5 + 4 * r1]4924
+ vinserti32x4 m4, [r4 + 2 * r1], 14925
+ vinserti32x4 m4, [r4 + 4 * r1], 24926
+ vinserti32x4 m4, [r8 + 2 * r1], 34927
+ punpcklbw m6, m5, m44928
+ pmaddubsw m6, m114929
+ punpckhbw m5, m44930
+ pmaddubsw m5, m114931
+4932
+ paddw m14, m64933
+ paddw m15, m54934
+4935
+ paddw m0, m124936
+ paddw m1, m134937
+ paddw m2, m144938
+ paddw m3, m154939
+%ifidn %1,pp4940
+ pmulhrsw m0, m74941
+ pmulhrsw m1, m74942
+ pmulhrsw m2, m74943
+ pmulhrsw m3, m74944
+4945
+ packuswb m0, m14946
+ packuswb m2, m34947
+ movu [r2], xm04948
+ movu [r2 + r3], xm24949
+ vextracti32x4 [r2 + 2 * r3], m0, 14950
+ vextracti32x4 [r2 + r7], m2, 14951
+ lea r2, [r2 + 4 * r3]4952
+ vextracti32x4 [r2], m0, 24953
+ vextracti32x4 [r2 + r3], m2, 24954
+ vextracti32x4 [r2 + 2 * r3], m0, 34955
+ vextracti32x4 [r2 + r7], m2, 34956
+%else4957
+ psubw m0, m74958
+ psubw m1, m74959
+ mova m12, m164960
+ mova m13, m174961
+ vpermi2q m12, m0, m14962
+ vpermi2q m13, m0, m14963
+ movu [r2], ym124964
+ vextracti32x8 [r2 + 2 * r3], m12, 14965
+4966
+ psubw m2, m74967
+ psubw m3, m74968
+ mova m14, m164969
+ mova m15, m174970
+ vpermi2q m14, m2, m34971
+ vpermi2q m15, m2, m34972
+ movu [r2 + r3], ym144973
+ vextracti32x8 [r2 + r7], m14, 14974
+ lea r2, [r2 + 4 * r3]4975
+4976
+ movu [r2], ym134977
+ movu [r2 + r3], ym154978
+ vextracti32x8 [r2 + 2 * r3], m13, 14979
+ vextracti32x8 [r2 + r7], m15, 14980
+%endif4981
+%endmacro4982
+;-----------------------------------------------------------------------------------------------------------------4983
+; void interp_4tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)4984
+;-----------------------------------------------------------------------------------------------------------------4985
+%macro FILTER_VERT_LUMA_16xN_AVX512 24986
+INIT_ZMM avx5124987
+cglobal interp_8tap_vert_%1_16x%2, 5, 9, 184988
+ mov r4d, r4m4989
+ shl r4d, 84990
+%ifdef PIC4991
+ lea r5, [tab_LumaCoeffVer_32_avx512]4992
+ mova m8, [r5 + r4]4993
+ mova m9, [r5 + r4 + 1 * mmsize]4994
+ mova m10, [r5 + r4 + 2 * mmsize]4995
+ mova m11, [r5 + r4 + 3 * mmsize]4996
+%else4997
+ mova m8, [tab_LumaCoeffVer_32_avx512 + r4]4998
+ mova m9, [tab_LumaCoeffVer_32_avx512 + r4 + 1 * mmsize]4999
+ mova m10, [tab_LumaCoeffVer_32_avx512 + r4 + 2 * mmsize]5000
+ mova m11, [tab_LumaCoeffVer_32_avx512 + r4 + 3 * mmsize]5001
+%endif5002
+%ifidn %1, pp5003
+ vbroadcasti32x8 m7, [pw_512]5004
+%else5005
+ shl r3d, 15006
+ vbroadcasti32x8 m7, [pw_2000]5007
+ mova m16, [interp4_vps_store1_avx512]5008
+ mova m17, [interp4_vps_store2_avx512]5009
+%endif5010
+5011
+ lea r6, [3 * r1]5012
+ lea r7, [3 * r3]5013
+ sub r0, r65014
+5015
+%rep %2/8 - 15016
+ PROCESS_LUMA_VERT_16x8_AVX512 %15017
+ lea r0, [r4]5018
+ lea r2, [r2 + 4 * r3]5019
+%endrep5020
+ PROCESS_LUMA_VERT_16x8_AVX512 %15021
+ RET5022
+%endmacro5023
+5024
+%if ARCH_X86_645025
+ FILTER_VERT_LUMA_16xN_AVX512 pp, 85026
+ FILTER_VERT_LUMA_16xN_AVX512 pp, 165027
+ FILTER_VERT_LUMA_16xN_AVX512 pp, 325028
+ FILTER_VERT_LUMA_16xN_AVX512 pp, 645029
+5030
+ FILTER_VERT_LUMA_16xN_AVX512 ps, 85031
+ FILTER_VERT_LUMA_16xN_AVX512 ps, 165032
+ FILTER_VERT_LUMA_16xN_AVX512 ps, 325033
+ FILTER_VERT_LUMA_16xN_AVX512 ps, 645034
+%endif5035
+%macro PROCESS_LUMA_VERT_32x4_AVX512 15036
+ lea r5, [r0 + 4 * r1]5037
+ movu ym1, [r0]5038
+ vinserti32x8 m1, [r0 + 2 * r1], 15039
+ movu ym3, [r0 + r1]5040
+ vinserti32x8 m3, [r0 + r6], 15041
+ punpcklbw m0, m1, m35042
+ pmaddubsw m0, m85043
+ punpckhbw m1, m35044
+ pmaddubsw m1, m85045
+5046
+ movu ym4, [r0 + 2 * r1]5047
+ vinserti32x8 m4, [r0 + 4 * r1], 15048
+ punpcklbw m2, m3, m45049
+ pmaddubsw m2, m85050
+ punpckhbw m3, m45051
+ pmaddubsw m3, m85052
+5053
+ movu ym5, [r0 + r6]5054
+ vinserti32x8 m5, [r5 + r1], 15055
+ punpcklbw m6, m4, m55056
+ pmaddubsw m6, m95057
+ punpckhbw m4, m55058
+ pmaddubsw m4, m95059
+5060
+ paddw m0, m65061
+ paddw m1, m45062
+5063
+ movu ym4, [r0 + 4 * r1]5064
+ vinserti32x8 m4, [r5 + 2 * r1], 15065
+ punpcklbw m6, m5, m45066
+ pmaddubsw m6, m95067
+ punpckhbw m5, m45068
+ pmaddubsw m5, m95069
+5070
+ paddw m2, m65071
+ paddw m3, m55072
+5073
+ lea r4, [r5 + 4 * r1]5074
+ movu ym15, [r5 + r1]5075
+ vinserti32x8 m15, [r5 + r6], 15076
+ punpcklbw m12, m4, m155077
+ pmaddubsw m12, m105078
+ punpckhbw m13, m4, m155079
+ pmaddubsw m13, m105080
+5081
+ movu ym4, [r5 + 2 * r1]5082
+ vinserti32x8 m4, [r5 + 4 * r1], 15083
+ punpcklbw m14, m15, m45084
+ pmaddubsw m14, m105085
+ punpckhbw m15, m45086
+ pmaddubsw m15, m105087
+5088
+ movu ym5, [r5 + r6]5089
+ vinserti32x8 m5, [r4 + r1], 15090
+ punpcklbw m6, m4, m55091
+ pmaddubsw m6, m115092
+ punpckhbw m4, m55093
+ pmaddubsw m4, m115094
+5095
+ paddw m12, m65096
+ paddw m13, m45097
+5098
+ movu ym4, [r5 + 4 * r1]5099
+ vinserti32x8 m4, [r4 + 2 * r1], 15100
+ punpcklbw m6, m5, m45101
+ pmaddubsw m6, m115102
+ punpckhbw m5, m45103
+ pmaddubsw m5, m115104
+5105
+ paddw m14, m65106
+ paddw m15, m55107
+5108
+ paddw m0, m125109
+ paddw m1, m135110
+ paddw m2, m145111
+ paddw m3, m155112
+%ifidn %1,pp5113
+ pmulhrsw m0, m75114
+ pmulhrsw m1, m75115
+ pmulhrsw m2, m75116
+ pmulhrsw m3, m75117
+5118
+ packuswb m0, m15119
+ packuswb m2, m35120
+ movu [r2], ym05121
+ movu [r2 + r3], ym25122
+ vextracti32x8 [r2 + 2 * r3], m0, 15123
+ vextracti32x8 [r2 + r7], m2, 15124
+%else5125
+ psubw m0, m75126
+ psubw m1, m75127
+ mova m12, m165128
+ mova m13, m175129
+ vpermi2q m12, m0, m15130
+ vpermi2q m13, m0, m15131
+ movu [r2], m125132
+ movu [r2 + 2 * r3], m135133
+5134
+ psubw m2, m75135
+ psubw m3, m75136
+ mova m14, m165137
+ mova m15, m175138
+ vpermi2q m14, m2, m35139
+ vpermi2q m15, m2, m35140
+ movu [r2 + r3], m145141
+ movu [r2 + r7], m155142
+%endif5143
+%endmacro5144
+;-----------------------------------------------------------------------------------------------------------------5145
+; void interp_4tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)5146
+;-----------------------------------------------------------------------------------------------------------------5147
+%macro FILTER_VERT_LUMA_32xN_AVX512 25148
+INIT_ZMM avx5125149
+cglobal interp_8tap_vert_%1_32x%2, 5, 8, 185150
+ mov r4d, r4m5151
+ shl r4d, 85152
+%ifdef PIC5153
+ lea r5, [tab_LumaCoeffVer_32_avx512]5154
+ mova m8, [r5 + r4]5155
+ mova m9, [r5 + r4 + 1 * mmsize]5156
+ mova m10, [r5 + r4 + 2 * mmsize]5157
+ mova m11, [r5 + r4 + 3 * mmsize]5158
+%else5159
+ mova m8, [tab_LumaCoeffVer_32_avx512 + r4]5160
+ mova m9, [tab_LumaCoeffVer_32_avx512 + r4 + 1 * mmsize]5161
+ mova m10, [tab_LumaCoeffVer_32_avx512 + r4 + 2 * mmsize]5162
+ mova m11, [tab_LumaCoeffVer_32_avx512 + r4 + 3 * mmsize]5163
+%endif5164
+%ifidn %1, pp5165
+ vbroadcasti32x8 m7, [pw_512]5166
+%else5167
+ shl r3d, 15168
+ vbroadcasti32x8 m7, [pw_2000]5169
+ mova m16, [interp4_vps_store1_avx512]5170
+ mova m17, [interp4_vps_store2_avx512]5171
+%endif5172
+5173
+ lea r6, [3 * r1]5174
+ lea r7, [3 * r3]5175
+ sub r0, r65176
+5177
+%rep %2/4 - 15178
+ PROCESS_LUMA_VERT_32x4_AVX512 %15179
+ lea r0, [r0 + 4 * r1]5180
+ lea r2, [r2 + 4 * r3]5181
+%endrep5182
+ PROCESS_LUMA_VERT_32x4_AVX512 %15183
+ RET5184
+%endmacro5185
+5186
+%if ARCH_X86_645187
+ FILTER_VERT_LUMA_32xN_AVX512 pp, 85188
+ FILTER_VERT_LUMA_32xN_AVX512 pp, 165189
+ FILTER_VERT_LUMA_32xN_AVX512 pp, 245190
+ FILTER_VERT_LUMA_32xN_AVX512 pp, 325191
+ FILTER_VERT_LUMA_32xN_AVX512 pp, 645192
+5193
+ FILTER_VERT_LUMA_32xN_AVX512 ps, 85194
+ FILTER_VERT_LUMA_32xN_AVX512 ps, 165195
+ FILTER_VERT_LUMA_32xN_AVX512 ps, 245196
+ FILTER_VERT_LUMA_32xN_AVX512 ps, 325197
+ FILTER_VERT_LUMA_32xN_AVX512 ps, 645198
+%endif5199
+%macro PROCESS_LUMA_VERT_48x8_AVX512 15200
+%ifidn %1, pp5201
+ PROCESS_LUMA_VERT_32x4_AVX512 pp5202
+%else5203
+ PROCESS_LUMA_VERT_32x4_AVX512 ps5204
+%endif5205
+ lea r8, [r4 + 4 * r1]5206
+ lea r9, [r2 + 4 * r3]5207
+ movu ym1, [r5]5208
+ vinserti32x8 m1, [r5 + 2 * r1], 15209
+ movu ym3, [r5 + r1]5210
+ vinserti32x8 m3, [r5 + r6], 15211
+ punpcklbw m0, m1, m35212
+ pmaddubsw m0, m85213
+ punpckhbw m1, m35214
+ pmaddubsw m1, m85215
+5216
+ movu ym4, [r5 + 2 * r1]5217
+ vinserti32x8 m4, [r5 + 4 * r1], 15218
+ punpcklbw m2, m3, m45219
+ pmaddubsw m2, m85220
+ punpckhbw m3, m45221
+ pmaddubsw m3, m85222
+5223
+ movu ym5, [r5 + r6]5224
+ vinserti32x8 m5, [r4 + r1], 15225
+ punpcklbw m6, m4, m55226
+ pmaddubsw m6, m95227
+ punpckhbw m4, m55228
+ pmaddubsw m4, m95229
+5230
+ paddw m0, m65231
+ paddw m1, m45232
+5233
+ movu ym4, [r5 + 4 * r1]5234
+ vinserti32x8 m4, [r4 + 2 * r1], 15235
+ punpcklbw m6, m5, m45236
+ pmaddubsw m6, m95237
+ punpckhbw m5, m45238
+ pmaddubsw m5, m95239
+5240
+ paddw m2, m65241
+ paddw m3, m55242
+5243
+ movu ym15, [r4 + r1]5244
+ vinserti32x8 m15, [r4 + r6], 15245
+ punpcklbw m12, m4, m155246
+ pmaddubsw m12, m105247
+ punpckhbw m13, m4, m155248
+ pmaddubsw m13, m105249
+5250
+ movu ym4, [r4 + 2 * r1]5251
+ vinserti32x8 m4, [r4 + 4 * r1], 15252
+ punpcklbw m14, m15, m45253
+ pmaddubsw m14, m105254
+ punpckhbw m15, m45255
+ pmaddubsw m15, m105256
+5257
+ movu ym5, [r4 + r6]5258
+ vinserti32x8 m5, [r8 + r1], 15259
+ punpcklbw m6, m4, m55260
+ pmaddubsw m6, m115261
+ punpckhbw m4, m55262
+ pmaddubsw m4, m115263
+5264
+ paddw m12, m65265
+ paddw m13, m45266
+5267
+ movu ym4, [r4 + 4 * r1]5268
+ vinserti32x8 m4, [r8 + 2 * r1], 15269
+ punpcklbw m6, m5, m45270
+ pmaddubsw m6, m115271
+ punpckhbw m5, m45272
+ pmaddubsw m5, m115273
+5274
+ paddw m14, m65275
+ paddw m15, m55276
+5277
+ paddw m0, m125278
+ paddw m1, m135279
+ paddw m2, m145280
+ paddw m3, m155281
+%ifidn %1,pp5282
+ pmulhrsw m0, m75283
+ pmulhrsw m1, m75284
+ pmulhrsw m2, m75285
+ pmulhrsw m3, m75286
+ packuswb m0, m15287
+ packuswb m2, m35288
+5289
+ movu [r9], ym05290
+ movu [r9 + r3], ym25291
+ vextracti32x8 [r9 + 2 * r3], m0, 15292
+ vextracti32x8 [r9 + r7], m2, 15293
+%else5294
+ psubw m0, m75295
+ psubw m1, m75296
+ mova m12, m165297
+ mova m13, m175298
+ vpermi2q m12, m0, m15299
+ vpermi2q m13, m0, m15300
+ movu [r9], m125301
+ movu [r9 + 2 * r3], m135302
+5303
+ psubw m2, m75304
+ psubw m3, m75305
+ mova m14, m165306
+ mova m15, m175307
+ vpermi2q m14, m2, m35308
+ vpermi2q m15, m2, m35309
+ movu [r9 + r3], m145310
+ movu [r9 + r7], m155311
+%endif5312
+ movu xm1, [r0 + mmsize/2]5313
+ vinserti32x4 m1, [r0 + 2 * r1 + mmsize/2], 15314
+ vinserti32x4 m1, [r5 + mmsize/2], 25315
+ vinserti32x4 m1, [r5 + 2 * r1 + mmsize/2], 35316
+ movu xm3, [r0 + r1 + mmsize/2]5317
+ vinserti32x4 m3, [r0 + r6 + mmsize/2], 15318
+ vinserti32x4 m3, [r5 + r1 + mmsize/2], 25319
+ vinserti32x4 m3, [r5 + r6 + mmsize/2], 35320
+ punpcklbw m0, m1, m35321
+ pmaddubsw m0, m85322
+ punpckhbw m1, m35323
+ pmaddubsw m1, m85324
+5325
+ movu xm4, [r0 + 2 * r1 + mmsize/2]5326
+ vinserti32x4 m4, [r0 + 4 * r1 + mmsize/2], 15327
+ vinserti32x4 m4, [r5 + 2 * r1 + mmsize/2], 25328
+ vinserti32x4 m4, [r5 + 4 * r1 + mmsize/2], 35329
+ punpcklbw m2, m3, m45330
+ pmaddubsw m2, m85331
+ punpckhbw m3, m45332
+ pmaddubsw m3, m85333
+5334
+ movu xm5, [r0 + r6 + mmsize/2]5335
+ vinserti32x4 m5, [r5 + r1 + mmsize/2], 15336
+ vinserti32x4 m5, [r5 + r6 + mmsize/2], 25337
+ vinserti32x4 m5, [r4 + r1 + mmsize/2], 35338
+ punpcklbw m6, m4, m55339
+ pmaddubsw m6, m95340
+ punpckhbw m4, m55341
+ pmaddubsw m4, m95342
+5343
+ paddw m0, m65344
+ paddw m1, m45345
+5346
+ movu xm4, [r0 + 4 * r1 + mmsize/2]5347
+ vinserti32x4 m4, [r5 + 2 * r1 + mmsize/2], 15348
+ vinserti32x4 m4, [r5 + 4 * r1 + mmsize/2], 25349
+ vinserti32x4 m4, [r4 + 2 * r1 + mmsize/2], 35350
+ punpcklbw m6, m5, m45351
+ pmaddubsw m6, m95352
+ punpckhbw m5, m45353
+ pmaddubsw m5, m95354
+5355
+ paddw m2, m65356
+ paddw m3, m55357
+5358
+ movu xm15, [r5 + r1 + mmsize/2]5359
+ vinserti32x4 m15, [r5 + r6 + mmsize/2], 15360
+ vinserti32x4 m15, [r4 + r1 + mmsize/2], 25361
+ vinserti32x4 m15, [r4 + r6 + mmsize/2], 35362
+ punpcklbw m12, m4, m155363
+ pmaddubsw m12, m105364
+ punpckhbw m13, m4, m155365
+ pmaddubsw m13, m105366
+5367
+ movu xm4, [r5 + 2 * r1 + mmsize/2]5368
+ vinserti32x4 m4, [r5 + 4 * r1 + mmsize/2], 15369
+ vinserti32x4 m4, [r4 + 2 * r1 + mmsize/2], 25370
+ vinserti32x4 m4, [r4 + 4 * r1 + mmsize/2], 35371
+ punpcklbw m14, m15, m45372
+ pmaddubsw m14, m105373
+ punpckhbw m15, m45374
+ pmaddubsw m15, m105375
+5376
+ movu xm5, [r5 + r6 + mmsize/2]5377
+ vinserti32x4 m5, [r4 + r1 + mmsize/2], 15378
+ vinserti32x4 m5, [r4 + r6 + mmsize/2], 25379
+ vinserti32x4 m5, [r8 + r1 + mmsize/2], 35380
+ punpcklbw m6, m4, m55381
+ pmaddubsw m6, m115382
+ punpckhbw m4, m55383
+ pmaddubsw m4, m115384
+5385
+ paddw m12, m65386
+ paddw m13, m45387
+5388
+ movu xm4, [r5 + 4 * r1 + mmsize/2]5389
+ vinserti32x4 m4, [r4 + 2 * r1 + mmsize/2], 15390
+ vinserti32x4 m4, [r4 + 4 * r1 + mmsize/2], 25391
+ vinserti32x4 m4, [r8 + 2 * r1 + mmsize/2], 35392
+ punpcklbw m6, m5, m45393
+ pmaddubsw m6, m115394
+ punpckhbw m5, m45395
+ pmaddubsw m5, m115396
+5397
+ paddw m14, m65398
+ paddw m15, m55399
+5400
+ paddw m0, m125401
+ paddw m1, m135402
+ paddw m2, m145403
+ paddw m3, m155404
+%ifidn %1, pp5405
+ pmulhrsw m0, m75406
+ pmulhrsw m1, m75407
+ pmulhrsw m2, m75408
+ pmulhrsw m3, m75409
+5410
+ packuswb m0, m15411
+ packuswb m2, m35412
+ movu [r2 + mmsize/2], xm05413
+ movu [r2 + r3 + mmsize/2], xm25414
+ vextracti32x4 [r2 + 2 * r3 + mmsize/2], m0, 15415
+ vextracti32x4 [r2 + r7 + mmsize/2], m2, 15416
+ lea r2, [r2 + 4 * r3]5417
+ vextracti32x4 [r2 + mmsize/2], m0, 25418
+ vextracti32x4 [r2 + r3 + mmsize/2], m2, 25419
+ vextracti32x4 [r2 + 2 * r3 + mmsize/2], m0, 35420
+ vextracti32x4 [r2 + r7 + mmsize/2], m2, 35421
+%else5422
+ psubw m0, m75423
+ psubw m1, m75424
+ mova m12, m165425
+ mova m13, m175426
+ vpermi2q m12, m0, m15427
+ vpermi2q m13, m0, m15428
+ movu [r2 + mmsize], ym125429
+ vextracti32x8 [r2 + 2 * r3 + mmsize], m12, 15430
+5431
+ psubw m2, m75432
+ psubw m3, m75433
+ mova m14, m165434
+ mova m15, m175435
+ vpermi2q m14, m2, m35436
+ vpermi2q m15, m2, m35437
+ movu [r2 + r3 + mmsize], ym145438
+ vextracti32x8 [r2 + r7 + mmsize], m14, 15439
+ lea r2, [r2 + 4 * r3]5440
+5441
+ movu [r2 + mmsize], ym135442
+ movu [r2 + r3 + mmsize], ym155443
+ vextracti32x8 [r2 + 2 * r3 + mmsize], m13, 15444
+ vextracti32x8 [r2 + r7 + mmsize], m15, 15445
+%endif5446
+%endmacro5447
+;-----------------------------------------------------------------------------------------------------------------5448
+; void interp_8tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)5449
+;-----------------------------------------------------------------------------------------------------------------5450
+%macro FILTER_VERT_LUMA_48x64_AVX512 15451
+INIT_ZMM avx5125452
+cglobal interp_8tap_vert_%1_48x64, 5, 10, 185453
+ mov r4d, r4m5454
+ shl r4d, 85455
+5456
+%ifdef PIC5457
+ lea r5, [tab_LumaCoeffVer_32_avx512]5458
+ mova m8, [r5 + r4]5459
+ mova m9, [r5 + r4 + 1 * mmsize]5460
+ mova m10, [r5 + r4 + 2 * mmsize]5461
+ mova m11, [r5 + r4 + 3 * mmsize]5462
+%else5463
+ mova m8, [tab_LumaCoeffVer_32_avx512 + r4]5464
+ mova m9, [tab_LumaCoeffVer_32_avx512 + r4 + 1 * mmsize]5465
+ mova m10, [tab_LumaCoeffVer_32_avx512 + r4 + 2 * mmsize]5466
+ mova m11, [tab_LumaCoeffVer_32_avx512 + r4 + 3 * mmsize]5467
+%endif5468
+%ifidn %1, pp5469
+ vbroadcasti32x8 m7, [pw_512]5470
+%else5471
+ shl r3d, 15472
+ vbroadcasti32x8 m7, [pw_2000]5473
+ mova m16, [interp4_vps_store1_avx512]5474
+ mova m17, [interp4_vps_store2_avx512]5475
+%endif5476
+5477
+ lea r6, [3 * r1]5478
+ lea r7, [3 * r3]5479
+ sub r0, r65480
+5481
+%rep 75482
+ PROCESS_LUMA_VERT_48x8_AVX512 %15483
+ lea r0, [r4]5484
+ lea r2, [r2 + 4 * r3]5485
+%endrep5486
+ PROCESS_LUMA_VERT_48x8_AVX512 %15487
+ RET5488
+%endmacro5489
+5490
+%if ARCH_X86_645491
+ FILTER_VERT_LUMA_48x64_AVX512 pp5492
+ FILTER_VERT_LUMA_48x64_AVX512 ps5493
+%endif5494
+%macro PROCESS_LUMA_VERT_64x2_AVX512 15495
+ lea r5, [r0 + 4 * r1]5496
+ movu m1, [r0]5497
+ movu m3, [r0 + r1]5498
+ punpcklbw m0, m1, m35499
+ pmaddubsw m0, m85500
+ punpckhbw m1, m35501
+ pmaddubsw m1, m85502
+5503
+ movu m4, [r0 + 2 * r1]5504
+ punpcklbw m2, m3, m45505
+ pmaddubsw m2, m85506
+ punpckhbw m3, m45507
+ pmaddubsw m3, m85508
+5509
+ movu m5, [r0 + r6]5510
+ punpcklbw m6, m4, m55511
+ pmaddubsw m6, m95512
+ punpckhbw m4, m55513
+ pmaddubsw m4, m95514
+5515
+ paddw m0, m65516
+ paddw m1, m45517
+5518
+ movu m4, [r0 + 4 * r1]5519
+ punpcklbw m6, m5, m45520
+ pmaddubsw m6, m95521
+ punpckhbw m5, m45522
+ pmaddubsw m5, m95523
+5524
+ paddw m2, m65525
+ paddw m3, m55526
+5527
+ movu m15, [r5 + r1]5528
+ punpcklbw m12, m4, m155529
+ pmaddubsw m12, m105530
+ punpckhbw m13, m4, m155531
+ pmaddubsw m13, m105532
+5533
+ movu m4, [r5 + 2 * r1]5534
+ punpcklbw m14, m15, m45535
+ pmaddubsw m14, m105536
+ punpckhbw m15, m45537
+ pmaddubsw m15, m105538
+5539
+ movu m5, [r5 + r6]5540
+ punpcklbw m6, m4, m55541
+ pmaddubsw m6, m115542
+ punpckhbw m4, m55543
+ pmaddubsw m4, m115544
+5545
+ paddw m12, m65546
+ paddw m13, m45547
+5548
+ movu m4, [r5 + 4 * r1]5549
+ punpcklbw m6, m5, m45550
+ pmaddubsw m6, m115551
+ punpckhbw m5, m45552
+ pmaddubsw m5, m115553
+5554
+ paddw m14, m65555
+ paddw m15, m55556
+5557
+ paddw m0, m125558
+ paddw m1, m135559
+ paddw m2, m145560
+ paddw m3, m155561
+%ifidn %1,pp5562
+ pmulhrsw m0, m75563
+ pmulhrsw m1, m75564
+ pmulhrsw m2, m75565
+ pmulhrsw m3, m75566
+5567
+ packuswb m0, m15568
+ packuswb m2, m35569
+ movu [r2], m05570
+ movu [r2 + r3], m25571
+%else5572
+ psubw m0, m75573
+ psubw m1, m75574
+ mova m12, m165575
+ mova m13, m175576
+ vpermi2q m12, m0, m15577
+ vpermi2q m13, m0, m15578
+ movu [r2], m125579
+ movu [r2 + mmsize], m135580
+5581
+ psubw m2, m75582
+ psubw m3, m75583
+ mova m14, m165584
+ mova m15, m175585
+ vpermi2q m14, m2, m35586
+ vpermi2q m15, m2, m35587
+ movu [r2 + r3], m145588
+ movu [r2 + r3 + mmsize], m155589
+%endif5590
+%endmacro5591
+;-----------------------------------------------------------------------------------------------------------------5592
+; void interp_4tap_vert(int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)5593
+;-----------------------------------------------------------------------------------------------------------------5594
+%macro FILTER_VERT_LUMA_64xN_AVX512 25595
+INIT_ZMM avx5125596
+cglobal interp_8tap_vert_%1_64x%2, 5, 8, 185597
+ mov r4d, r4m5598
+ shl r4d, 85599
+%ifdef PIC5600
+ lea r5, [tab_LumaCoeffVer_32_avx512]5601
+ mova m8, [r5 + r4]5602
+ mova m9, [r5 + r4 + 1 * mmsize]5603
+ mova m10, [r5 + r4 + 2 * mmsize]5604
+ mova m11, [r5 + r4 + 3 * mmsize]5605
+%else5606
+ mova m8, [tab_LumaCoeffVer_32_avx512 + r4]5607
+ mova m9, [tab_LumaCoeffVer_32_avx512 + r4 + 1 * mmsize]5608
+ mova m10, [tab_LumaCoeffVer_32_avx512 + r4 + 2 * mmsize]5609
+ mova m11, [tab_LumaCoeffVer_32_avx512 + r4 + 3 * mmsize]5610
+%endif5611
+%ifidn %1, pp5612
+ vbroadcasti32x8 m7, [pw_512]5613
+%else5614
+ shl r3d, 15615
+ vbroadcasti32x8 m7, [pw_2000]5616
+ mova m16, [interp4_vps_store1_avx512]5617
+ mova m17, [interp4_vps_store2_avx512]5618
+%endif5619
+5620
+ lea r6, [3 * r1]5621
+ sub r0, r65622
+ lea r7, [3 * r3]5623
+5624
+%rep %2/2 - 15625
+ PROCESS_LUMA_VERT_64x2_AVX512 %15626
+ lea r0, [r0 + 2 * r1]5627
+ lea r2, [r2 + 2 * r3]5628
+%endrep5629
+ PROCESS_LUMA_VERT_64x2_AVX512 %15630
+ RET5631
+%endmacro5632
+5633
+%if ARCH_X86_645634
+FILTER_VERT_LUMA_64xN_AVX512 pp, 165635
+FILTER_VERT_LUMA_64xN_AVX512 pp, 325636
+FILTER_VERT_LUMA_64xN_AVX512 pp, 485637
+FILTER_VERT_LUMA_64xN_AVX512 pp, 645638
+5639
+FILTER_VERT_LUMA_64xN_AVX512 ps, 165640
+FILTER_VERT_LUMA_64xN_AVX512 ps, 325641
+FILTER_VERT_LUMA_64xN_AVX512 ps, 485642
+FILTER_VERT_LUMA_64xN_AVX512 ps, 645643
+%endif5644
+;-------------------------------------------------------------------------------------------------------------5645
+;avx512 luma_vpp and luma_vps code end5646
+;-------------------------------------------------------------------------------------------------------------5647
+;-------------------------------------------------------------------------------------------------------------5648
+;ipfilter_luma_avx512 code end5649
+;-------------------------------------------------------------------------------------------------------------5650
\ No newline at end of file5651
x265_2.7.tar.gz/source/common/x86/ipfilter8.h -> x265_2.9.tar.gz/source/common/x86/ipfilter8.h
Changed
16
1
2
FUNCDEF_PU(void, interp_8tap_vert_ss, cpu, const int16_t* src, intptr_t srcStride, int16_t* dst, intptr_t dstStride, int coeffIdx); \3
FUNCDEF_PU(void, interp_8tap_hv_pp, cpu, const pixel* src, intptr_t srcStride, pixel* dst, intptr_t dstStride, int idxX, int idxY); \4
FUNCDEF_CHROMA_PU(void, filterPixelToShort, cpu, const pixel* src, intptr_t srcStride, int16_t* dst, intptr_t dstStride); \5
+ FUNCDEF_CHROMA_PU(void, filterPixelToShort_aligned, cpu, const pixel* src, intptr_t srcStride, int16_t* dst, intptr_t dstStride); \6
FUNCDEF_CHROMA_PU(void, interp_4tap_horiz_pp, cpu, const pixel* src, intptr_t srcStride, pixel* dst, intptr_t dstStride, int coeffIdx); \7
FUNCDEF_CHROMA_PU(void, interp_4tap_horiz_ps, cpu, const pixel* src, intptr_t srcStride, int16_t* dst, intptr_t dstStride, int coeffIdx, int isRowExt); \8
FUNCDEF_CHROMA_PU(void, interp_4tap_vert_pp, cpu, const pixel* src, intptr_t srcStride, pixel* dst, intptr_t dstStride, int coeffIdx); \9
10
SETUP_FUNC_DEF(sse3);11
SETUP_FUNC_DEF(sse4);12
SETUP_FUNC_DEF(avx2);13
+SETUP_FUNC_DEF(avx512);14
15
#endif // ifndef X265_IPFILTER8_H16
x265_2.7.tar.gz/source/common/x86/loopfilter.asm -> x265_2.9.tar.gz/source/common/x86/loopfilter.asm
Changed
50
1
2
;============================================================================================================3
INIT_XMM sse44
%if HIGH_BIT_DEPTH5
+%if ARCH_X86_646
cglobal saoCuOrgE0, 4,5,97
mov r4d, r4m8
movh m6, [r1]9
10
sub r4d, 1611
jnz .loopH12
RET13
-14
+%endif15
%else ; HIGH_BIT_DEPTH == 116
17
cglobal saoCuOrgE0, 5, 5, 8, rec, offsetEo, lcuWidth, signLeft, stride18
19
20
INIT_YMM avx221
%if HIGH_BIT_DEPTH22
+%if ARCH_X86_6423
cglobal saoCuOrgE0, 4,4,924
vbroadcasti128 m6, [r1]25
movzx r1d, byte [r3]26
27
dec r2d28
jnz .loop29
RET30
+%endif31
%else ; HIGH_BIT_DEPTH32
cglobal saoCuOrgE0, 5, 5, 7, rec, offsetEo, lcuWidth, signLeft, stride33
34
35
RET36
%endif37
38
+%if ARCH_X86_6439
INIT_YMM avx240
%if HIGH_BIT_DEPTH41
cglobal saoCuOrgB0, 5,7,842
43
.end:44
RET45
%endif46
+%endif47
48
;============================================================================================================49
; void calSign(int8_t *dst, const Pixel *src1, const Pixel *src2, const int width)50
x265_2.7.tar.gz/source/common/x86/mc-a.asm -> x265_2.9.tar.gz/source/common/x86/mc-a.asm
Changed
1841
1
2
%error Unsupport bit depth!3
%endif4
5
-SECTION_RODATA 326
+SECTION_RODATA 647
8
-ch_shuf: times 2 db 0,2,2,4,4,6,6,8,1,3,3,5,5,7,7,99
-ch_shuf_adj: times 8 db 010
- times 8 db 211
- times 8 db 412
- times 8 db 613
+ALIGN 6414
+const shuf_avx512, dq 0, 2, 4, 6, 1, 3, 5, 715
16
SECTION .text17
18
19
;------------------------------------------------------------------------------20
; avx2 asm for addAvg high_bit_depth21
;------------------------------------------------------------------------------22
+%if ARCH_X86_6423
INIT_YMM avx224
cglobal addAvg_8x2, 6,6,2, pSrc0, pSrc1, pDst, iStride0, iStride1, iDstStride25
movu xm0, [r0]26
27
movu [r2], xm028
movu [r2 + r5], xm229
RET30
+%endif31
32
%macro ADDAVG_W8_H4_AVX2 133
cglobal addAvg_8x%1, 6,7,6, pSrc0, pSrc1, pDst, iStride0, iStride1, iDstStride34
35
RET36
%endmacro37
38
+%if ARCH_X86_6439
ADDAVG_W8_H4_AVX2 440
ADDAVG_W8_H4_AVX2 841
ADDAVG_W8_H4_AVX2 1242
ADDAVG_W8_H4_AVX2 1643
ADDAVG_W8_H4_AVX2 3244
ADDAVG_W8_H4_AVX2 6445
+%endif46
47
+%if ARCH_X86_6448
cglobal addAvg_12x16, 6,7,6, pSrc0, pSrc1, pDst, iStride0, iStride1, iDstStride49
mova m4, [pw_ %+ ADDAVG_ROUND]50
mova m5, [pw_pixel_max]51
52
dec r6d53
jnz .loop54
RET55
+%endif56
57
%macro ADDAVG_W16_H4_AVX2 158
cglobal addAvg_16x%1, 6,7,6, pSrc0, pSrc1, pDst, iStride0, iStride1, iDstStride59
60
RET61
%endmacro62
63
+%if ARCH_X86_6464
ADDAVG_W16_H4_AVX2 465
ADDAVG_W16_H4_AVX2 866
ADDAVG_W16_H4_AVX2 1267
68
ADDAVG_W16_H4_AVX2 2469
ADDAVG_W16_H4_AVX2 3270
ADDAVG_W16_H4_AVX2 6471
+%endif72
73
+%if ARCH_X86_6474
cglobal addAvg_24x32, 6,7,6, pSrc0, pSrc1, pDst, iStride0, iStride1, iDstStride75
mova m4, [pw_ %+ ADDAVG_ROUND]76
mova m5, [pw_pixel_max]77
78
dec r6d79
jnz .loop80
RET81
+%endif82
83
%macro ADDAVG_W32_H2_AVX2 184
cglobal addAvg_32x%1, 6,7,6, pSrc0, pSrc1, pDst, iStride0, iStride1, iDstStride85
86
RET87
%endmacro88
89
+%if ARCH_X86_6490
ADDAVG_W32_H2_AVX2 891
ADDAVG_W32_H2_AVX2 1692
ADDAVG_W32_H2_AVX2 2493
ADDAVG_W32_H2_AVX2 3294
ADDAVG_W32_H2_AVX2 4895
ADDAVG_W32_H2_AVX2 6496
+%endif97
98
+%if ARCH_X86_6499
cglobal addAvg_48x64, 6,7,6, pSrc0, pSrc1, pDst, iStride0, iStride1, iDstStride100
mova m4, [pw_ %+ ADDAVG_ROUND]101
mova m5, [pw_pixel_max]102
103
dec r6d104
jnz .loop105
RET106
+%endif107
108
%macro ADDAVG_W64_H1_AVX2 1109
cglobal addAvg_64x%1, 6,7,6, pSrc0, pSrc1, pDst, iStride0, iStride1, iDstStride110
111
RET112
%endmacro113
114
+%if ARCH_X86_64115
ADDAVG_W64_H1_AVX2 16116
ADDAVG_W64_H1_AVX2 32117
ADDAVG_W64_H1_AVX2 48118
ADDAVG_W64_H1_AVX2 64119
+%endif120
+;-----------------------------------------------------------------------------121
+;addAvg avx512 high bit depth code start122
+;-----------------------------------------------------------------------------123
+%macro PROCESS_ADDAVG_16x4_HBD_AVX512 0124
+ movu ym0, [r0]125
+ vinserti32x8 m0, [r0 + r3], 1126
+ movu ym1, [r1]127
+ vinserti32x8 m1, [r1 + r4], 1128
+129
+ paddw m0, m1130
+ pmulhrsw m0, m3131
+ paddw m0, m4132
+ pmaxsw m0, m2133
+ pminsw m0, m5134
+135
+ movu [r2], ym0136
+ vextracti32x8 [r2 + r5], m0, 1137
+138
+ movu ym0, [r0 + 2 * r3]139
+ vinserti32x8 m0, [r0 + r6], 1140
+ movu ym1, [r1 + 2 * r4]141
+ vinserti32x8 m1, [r1 + r7], 1142
+143
+ paddw m0, m1144
+ pmulhrsw m0, m3145
+ paddw m0, m4146
+ pmaxsw m0, m2147
+ pminsw m0, m5148
+149
+ movu [r2 + 2 * r5], ym0150
+ vextracti32x8 [r2 + r8], m0, 1151
+%endmacro152
+153
+%macro PROCESS_ADDAVG_32x4_HBD_AVX512 0154
+ movu m0, [r0]155
+ movu m1, [r1]156
+ paddw m0, m1157
+ pmulhrsw m0, m3158
+ paddw m0, m4159
+ pmaxsw m0, m2160
+ pminsw m0, m5161
+ movu [r2], m0162
+163
+ movu m0, [r0 + r3]164
+ movu m1, [r1 + r4]165
+ paddw m0, m1166
+ pmulhrsw m0, m3167
+ paddw m0, m4168
+ pmaxsw m0, m2169
+ pminsw m0, m5170
+ movu [r2 + r5], m0171
+172
+ movu m0, [r0 + 2 * r3]173
+ movu m1, [r1 + 2 * r4]174
+ paddw m0, m1175
+ pmulhrsw m0, m3176
+ paddw m0, m4177
+ pmaxsw m0, m2178
+ pminsw m0, m5179
+ movu [r2 + 2 * r5], m0180
+181
+ movu m0, [r0 + r6]182
+ movu m1, [r1 + r7]183
+ paddw m0, m1184
+ pmulhrsw m0, m3185
+ paddw m0, m4186
+ pmaxsw m0, m2187
+ pminsw m0, m5188
+ movu [r2 + r8], m0189
+%endmacro190
+191
+%macro PROCESS_ADDAVG_64x4_HBD_AVX512 0192
+ movu m0, [r0]193
+ movu m1, [r1]194
+ paddw m0, m1195
+ pmulhrsw m0, m3196
+ paddw m0, m4197
+ pmaxsw m0, m2198
+ pminsw m0, m5199
+ movu [r2], m0200
+201
+ movu m0, [r0 + mmsize]202
+ movu m1, [r1 + mmsize]203
+ paddw m0, m1204
+ pmulhrsw m0, m3205
+ paddw m0, m4206
+ pmaxsw m0, m2207
+ pminsw m0, m5208
+ movu [r2 + mmsize], m0209
+210
+ movu m0, [r0 + r3]211
+ movu m1, [r1 + r4]212
+ paddw m0, m1213
+ pmulhrsw m0, m3214
+ paddw m0, m4215
+ pmaxsw m0, m2216
+ pminsw m0, m5217
+ movu [r2 + r5], m0218
+219
+ movu m0, [r0 + r3 + mmsize]220
+ movu m1, [r1 + r4 + mmsize]221
+ paddw m0, m1222
+ pmulhrsw m0, m3223
+ paddw m0, m4224
+ pmaxsw m0, m2225
+ pminsw m0, m5226
+ movu [r2 + r5 + mmsize], m0227
+228
+ movu m0, [r0 + 2 * r3]229
+ movu m1, [r1 + 2 * r4]230
+ paddw m0, m1231
+ pmulhrsw m0, m3232
+ paddw m0, m4233
+ pmaxsw m0, m2234
+ pminsw m0, m5235
+ movu [r2 + 2 * r5], m0236
+237
+ movu m0, [r0 + 2 * r3 + mmsize]238
+ movu m1, [r1 + 2 * r4 + mmsize]239
+ paddw m0, m1240
+ pmulhrsw m0, m3241
+ paddw m0, m4242
+ pmaxsw m0, m2243
+ pminsw m0, m5244
+ movu [r2 + 2 * r5 + mmsize], m0245
+246
+ movu m0, [r0 + r6]247
+ movu m1, [r1 + r7]248
+ paddw m0, m1249
+ pmulhrsw m0, m3250
+ paddw m0, m4251
+ pmaxsw m0, m2252
+ pminsw m0, m5253
+ movu [r2 + r8], m0254
+255
+ movu m0, [r0 + r6 + mmsize]256
+ movu m1, [r1 + r7 + mmsize]257
+ paddw m0, m1258
+ pmulhrsw m0, m3259
+ paddw m0, m4260
+ pmaxsw m0, m2261
+ pminsw m0, m5262
+ movu [r2 + r8 + mmsize], m0263
+%endmacro264
+265
+%macro PROCESS_ADDAVG_48x4_HBD_AVX512 0266
+ movu m0, [r0]267
+ movu m1, [r1]268
+ paddw m0, m1269
+ pmulhrsw m0, m3270
+ paddw m0, m4271
+ pmaxsw m0, m2272
+ pminsw m0, m5273
+ movu [r2], m0274
+275
+ movu ym0, [r0 + mmsize]276
+ movu ym1, [r1 + mmsize]277
+ paddw ym0, ym1278
+ pmulhrsw ym0, ym3279
+ paddw ym0, ym4280
+ pmaxsw ym0, ym2281
+ pminsw ym0, ym5282
+ movu [r2 + mmsize], ym0283
+284
+ movu m0, [r0 + r3]285
+ movu m1, [r1 + r4]286
+ paddw m0, m1287
+ pmulhrsw m0, m3288
+ paddw m0, m4289
+ pmaxsw m0, m2290
+ pminsw m0, m5291
+ movu [r2 + r5], m0292
+293
+ movu ym0, [r0 + r3 + mmsize]294
+ movu ym1, [r1 + r4 + mmsize]295
+ paddw ym0, ym1296
+ pmulhrsw ym0, ym3297
+ paddw ym0, ym4298
+ pmaxsw ym0, ym2299
+ pminsw ym0, ym5300
+ movu [r2 + r5 + mmsize], ym0301
+302
+ movu m0, [r0 + 2 * r3]303
+ movu m1, [r1 + 2 * r4]304
+ paddw m0, m1305
+ pmulhrsw m0, m3306
+ paddw m0, m4307
+ pmaxsw m0, m2308
+ pminsw m0, m5309
+ movu [r2 + 2 * r5], m0310
+311
+ movu ym0, [r0 + 2 * r3 + mmsize]312
+ movu ym1, [r1 + 2 * r4 + mmsize]313
+ paddw ym0, ym1314
+ pmulhrsw ym0, ym3315
+ paddw ym0, ym4316
+ pmaxsw ym0, ym2317
+ pminsw ym0, ym5318
+ movu [r2 + 2 * r5 + mmsize], ym0319
+320
+ movu m0, [r0 + r6]321
+ movu m1, [r1 + r7]322
+ paddw m0, m1323
+ pmulhrsw m0, m3324
+ paddw m0, m4325
+ pmaxsw m0, m2326
+ pminsw m0, m5327
+ movu [r2 + r8], m0328
+329
+ movu ym0, [r0 + r6 + mmsize]330
+ movu ym1, [r1 + r7 + mmsize]331
+ paddw ym0, ym1332
+ pmulhrsw ym0, ym3333
+ paddw ym0, ym4334
+ pmaxsw ym0, ym2335
+ pminsw ym0, ym5336
+ movu [r2 + r8 + mmsize], ym0337
+%endmacro338
+;-----------------------------------------------------------------------------339
+;void addAvg (int16_t* src0, int16_t* src1, pixel* dst, intptr_t src0Stride, intptr_t src1Stride, intptr_t dstStride)340
+;-----------------------------------------------------------------------------341
+%if ARCH_X86_64342
+INIT_ZMM avx512343
+cglobal addAvg_16x4, 6,9,6344
+ vbroadcasti32x8 m4, [pw_ %+ ADDAVG_ROUND]345
+ vbroadcasti32x8 m5, [pw_pixel_max]346
+ vbroadcasti32x8 m3, [pw_ %+ ADDAVG_FACTOR]347
+ pxor m2, m2348
+ add r3, r3349
+ add r4, r4350
+ add r5, r5351
+ lea r6, [3 * r3]352
+ lea r7, [3 * r4]353
+ lea r8, [3 * r5]354
+ PROCESS_ADDAVG_16x4_HBD_AVX512355
+ RET356
+%endif357
+358
+%macro ADDAVG_W16_HBD_AVX512 1359
+INIT_ZMM avx512360
+cglobal addAvg_16x%1, 6,9,6361
+ vbroadcasti32x8 m4, [pw_ %+ ADDAVG_ROUND]362
+ vbroadcasti32x8 m5, [pw_pixel_max]363
+ vbroadcasti32x8 m3, [pw_ %+ ADDAVG_FACTOR]364
+ pxor m2, m2365
+ add r3, r3366
+ add r4, r4367
+ add r5, r5368
+ lea r6, [3 * r3]369
+ lea r7, [3 * r4]370
+ lea r8, [3 * r5]371
+372
+%rep %1/4 - 1373
+ PROCESS_ADDAVG_16x4_HBD_AVX512374
+ lea r2, [r2 + 4 * r5]375
+ lea r0, [r0 + 4 * r3]376
+ lea r1, [r1 + 4 * r4]377
+%endrep378
+ PROCESS_ADDAVG_16x4_HBD_AVX512379
+ RET380
+%endmacro381
+382
+%if ARCH_X86_64383
+ADDAVG_W16_HBD_AVX512 8384
+ADDAVG_W16_HBD_AVX512 12385
+ADDAVG_W16_HBD_AVX512 16386
+ADDAVG_W16_HBD_AVX512 24387
+ADDAVG_W16_HBD_AVX512 32388
+ADDAVG_W16_HBD_AVX512 64389
+%endif390
+391
+%macro ADDAVG_W32_HBD_AVX512 1392
+INIT_ZMM avx512393
+cglobal addAvg_32x%1, 6,9,6394
+ vbroadcasti32x8 m4, [pw_ %+ ADDAVG_ROUND]395
+ vbroadcasti32x8 m5, [pw_pixel_max]396
+ vbroadcasti32x8 m3, [pw_ %+ ADDAVG_FACTOR]397
+ pxor m2, m2398
+ add r3, r3399
+ add r4, r4400
+ add r5, r5401
+ lea r6, [3 * r3]402
+ lea r7, [3 * r4]403
+ lea r8, [3 * r5]404
+405
+%rep %1/4 - 1406
+ PROCESS_ADDAVG_32x4_HBD_AVX512407
+ lea r2, [r2 + 4 * r5]408
+ lea r0, [r0 + 4 * r3]409
+ lea r1, [r1 + 4 * r4]410
+%endrep411
+ PROCESS_ADDAVG_32x4_HBD_AVX512412
+ RET413
+%endmacro414
+415
+%if ARCH_X86_64416
+ADDAVG_W32_HBD_AVX512 8417
+ADDAVG_W32_HBD_AVX512 16418
+ADDAVG_W32_HBD_AVX512 24419
+ADDAVG_W32_HBD_AVX512 32420
+ADDAVG_W32_HBD_AVX512 48421
+ADDAVG_W32_HBD_AVX512 64422
+%endif423
+424
+%macro ADDAVG_W64_HBD_AVX512 1425
+INIT_ZMM avx512426
+cglobal addAvg_64x%1, 6,9,6427
+ vbroadcasti32x8 m4, [pw_ %+ ADDAVG_ROUND]428
+ vbroadcasti32x8 m5, [pw_pixel_max]429
+ vbroadcasti32x8 m3, [pw_ %+ ADDAVG_FACTOR]430
+ pxor m2, m2431
+ add r3, r3432
+ add r4, r4433
+ add r5, r5434
+ lea r6, [3 * r3]435
+ lea r7, [3 * r4]436
+ lea r8, [3 * r5]437
+438
+%rep %1/4 - 1439
+ PROCESS_ADDAVG_64x4_HBD_AVX512440
+ lea r2, [r2 + 4 * r5]441
+ lea r0, [r0 + 4 * r3]442
+ lea r1, [r1 + 4 * r4]443
+%endrep444
+ PROCESS_ADDAVG_64x4_HBD_AVX512445
+ RET446
+%endmacro447
+448
+%if ARCH_X86_64449
+ADDAVG_W64_HBD_AVX512 16450
+ADDAVG_W64_HBD_AVX512 32451
+ADDAVG_W64_HBD_AVX512 48452
+ADDAVG_W64_HBD_AVX512 64453
+%endif454
+455
+%if ARCH_X86_64456
+INIT_ZMM avx512457
+cglobal addAvg_48x64, 6,9,6458
+ vbroadcasti32x8 m4, [pw_ %+ ADDAVG_ROUND]459
+ vbroadcasti32x8 m5, [pw_pixel_max]460
+ vbroadcasti32x8 m3, [pw_ %+ ADDAVG_FACTOR]461
+ pxor m2, m2462
+ add r3, r3463
+ add r4, r4464
+ add r5, r5465
+ lea r6, [3 * r3]466
+ lea r7, [3 * r4]467
+ lea r8, [3 * r5]468
+469
+%rep 15470
+ PROCESS_ADDAVG_48x4_HBD_AVX512471
+ lea r2, [r2 + 4 * r5]472
+ lea r0, [r0 + 4 * r3]473
+ lea r1, [r1 + 4 * r4]474
+%endrep475
+ PROCESS_ADDAVG_48x4_HBD_AVX512476
+ RET477
+%endif478
+479
+%macro PROCESS_ADDAVG_ALIGNED_16x4_HBD_AVX512 0480
+ movu ym0, [r0]481
+ vinserti32x8 m0, [r0 + r3], 1482
+ movu ym1, [r1]483
+ vinserti32x8 m1, [r1 + r4], 1484
+485
+ paddw m0, m1486
+ pmulhrsw m0, m3487
+ paddw m0, m4488
+ pmaxsw m0, m2489
+ pminsw m0, m5490
+491
+ movu [r2], ym0492
+ vextracti32x8 [r2 + r5], m0, 1493
+494
+ movu ym0, [r0 + 2 * r3]495
+ vinserti32x8 m0, [r0 + r6], 1496
+ movu ym1, [r1 + 2 * r4]497
+ vinserti32x8 m1, [r1 + r7], 1498
+499
+ paddw m0, m1500
+ pmulhrsw m0, m3501
+ paddw m0, m4502
+ pmaxsw m0, m2503
+ pminsw m0, m5504
+505
+ movu [r2 + 2 * r5], ym0506
+ vextracti32x8 [r2 + r8], m0, 1507
+%endmacro508
+509
+%macro PROCESS_ADDAVG_ALIGNED_32x4_HBD_AVX512 0510
+ movu m0, [r0]511
+ movu m1, [r1]512
+ paddw m0, m1513
+ pmulhrsw m0, m3514
+ paddw m0, m4515
+ pmaxsw m0, m2516
+ pminsw m0, m5517
+ movu [r2], m0518
+519
+ movu m0, [r0 + r3]520
+ movu m1, [r1 + r4]521
+ paddw m0, m1522
+ pmulhrsw m0, m3523
+ paddw m0, m4524
+ pmaxsw m0, m2525
+ pminsw m0, m5526
+ movu [r2 + r5], m0527
+528
+ movu m0, [r0 + 2 * r3]529
+ movu m1, [r1 + 2 * r4]530
+ paddw m0, m1531
+ pmulhrsw m0, m3532
+ paddw m0, m4533
+ pmaxsw m0, m2534
+ pminsw m0, m5535
+ movu [r2 + 2 * r5], m0536
+537
+ movu m0, [r0 + r6]538
+ movu m1, [r1 + r7]539
+ paddw m0, m1540
+ pmulhrsw m0, m3541
+ paddw m0, m4542
+ pmaxsw m0, m2543
+ pminsw m0, m5544
+ movu [r2 + r8], m0545
+%endmacro546
+547
+%macro PROCESS_ADDAVG_ALIGNED_64x4_HBD_AVX512 0548
+ movu m0, [r0]549
+ movu m1, [r1]550
+ paddw m0, m1551
+ pmulhrsw m0, m3552
+ paddw m0, m4553
+ pmaxsw m0, m2554
+ pminsw m0, m5555
+ movu [r2], m0556
+557
+ movu m0, [r0 + mmsize]558
+ movu m1, [r1 + mmsize]559
+ paddw m0, m1560
+ pmulhrsw m0, m3561
+ paddw m0, m4562
+ pmaxsw m0, m2563
+ pminsw m0, m5564
+ movu [r2 + mmsize], m0565
+566
+ movu m0, [r0 + r3]567
+ movu m1, [r1 + r4]568
+ paddw m0, m1569
+ pmulhrsw m0, m3570
+ paddw m0, m4571
+ pmaxsw m0, m2572
+ pminsw m0, m5573
+ movu [r2 + r5], m0574
+575
+ movu m0, [r0 + r3 + mmsize]576
+ movu m1, [r1 + r4 + mmsize]577
+ paddw m0, m1578
+ pmulhrsw m0, m3579
+ paddw m0, m4580
+ pmaxsw m0, m2581
+ pminsw m0, m5582
+ movu [r2 + r5 + mmsize], m0583
+584
+ movu m0, [r0 + 2 * r3]585
+ movu m1, [r1 + 2 * r4]586
+ paddw m0, m1587
+ pmulhrsw m0, m3588
+ paddw m0, m4589
+ pmaxsw m0, m2590
+ pminsw m0, m5591
+ movu [r2 + 2 * r5], m0592
+593
+ movu m0, [r0 + 2 * r3 + mmsize]594
+ movu m1, [r1 + 2 * r4 + mmsize]595
+ paddw m0, m1596
+ pmulhrsw m0, m3597
+ paddw m0, m4598
+ pmaxsw m0, m2599
+ pminsw m0, m5600
+ movu [r2 + 2 * r5 + mmsize], m0601
+602
+ movu m0, [r0 + r6]603
+ movu m1, [r1 + r7]604
+ paddw m0, m1605
+ pmulhrsw m0, m3606
+ paddw m0, m4607
+ pmaxsw m0, m2608
+ pminsw m0, m5609
+ movu [r2 + r8], m0610
+611
+ movu m0, [r0 + r6 + mmsize]612
+ movu m1, [r1 + r7 + mmsize]613
+ paddw m0, m1614
+ pmulhrsw m0, m3615
+ paddw m0, m4616
+ pmaxsw m0, m2617
+ pminsw m0, m5618
+ movu [r2 + r8 + mmsize], m0619
+%endmacro620
+621
+%macro PROCESS_ADDAVG_ALIGNED_48x4_HBD_AVX512 0622
+ movu m0, [r0]623
+ movu m1, [r1]624
+ paddw m0, m1625
+ pmulhrsw m0, m3626
+ paddw m0, m4627
+ pmaxsw m0, m2628
+ pminsw m0, m5629
+ movu [r2], m0630
+631
+ movu ym0, [r0 + mmsize]632
+ movu ym1, [r1 + mmsize]633
+ paddw ym0, ym1634
+ pmulhrsw ym0, ym3635
+ paddw ym0, ym4636
+ pmaxsw ym0, ym2637
+ pminsw ym0, ym5638
+ movu [r2 + mmsize], ym0639
+640
+ movu m0, [r0 + r3]641
+ movu m1, [r1 + r4]642
+ paddw m0, m1643
+ pmulhrsw m0, m3644
+ paddw m0, m4645
+ pmaxsw m0, m2646
+ pminsw m0, m5647
+ movu [r2 + r5], m0648
+649
+ movu ym0, [r0 + r3 + mmsize]650
+ movu ym1, [r1 + r4 + mmsize]651
+ paddw ym0, ym1652
+ pmulhrsw ym0, ym3653
+ paddw ym0, ym4654
+ pmaxsw ym0, ym2655
+ pminsw ym0, ym5656
+ movu [r2 + r5 + mmsize], ym0657
+658
+ movu m0, [r0 + 2 * r3]659
+ movu m1, [r1 + 2 * r4]660
+ paddw m0, m1661
+ pmulhrsw m0, m3662
+ paddw m0, m4663
+ pmaxsw m0, m2664
+ pminsw m0, m5665
+ movu [r2 + 2 * r5], m0666
+667
+ movu ym0, [r0 + 2 * r3 + mmsize]668
+ movu ym1, [r1 + 2 * r4 + mmsize]669
+ paddw ym0, ym1670
+ pmulhrsw ym0, ym3671
+ paddw ym0, ym4672
+ pmaxsw ym0, ym2673
+ pminsw ym0, ym5674
+ movu [r2 + 2 * r5 + mmsize], ym0675
+676
+ movu m0, [r0 + r6]677
+ movu m1, [r1 + r7]678
+ paddw m0, m1679
+ pmulhrsw m0, m3680
+ paddw m0, m4681
+ pmaxsw m0, m2682
+ pminsw m0, m5683
+ movu [r2 + r8], m0684
+685
+ movu ym0, [r0 + r6 + mmsize]686
+ movu ym1, [r1 + r7 + mmsize]687
+ paddw ym0, ym1688
+ pmulhrsw ym0, ym3689
+ paddw ym0, ym4690
+ pmaxsw ym0, ym2691
+ pminsw ym0, ym5692
+ movu [r2 + r8 + mmsize], ym0693
+%endmacro694
+;-----------------------------------------------------------------------------695
+;void addAvg (int16_t* src0, int16_t* src1, pixel* dst, intptr_t src0Stride, intptr_t src1Stride, intptr_t dstStride)696
+;-----------------------------------------------------------------------------697
+%if ARCH_X86_64698
+INIT_ZMM avx512699
+cglobal addAvg_aligned_16x4, 6,9,6700
+ vbroadcasti32x8 m4, [pw_ %+ ADDAVG_ROUND]701
+ vbroadcasti32x8 m5, [pw_pixel_max]702
+ vbroadcasti32x8 m3, [pw_ %+ ADDAVG_FACTOR]703
+ pxor m2, m2704
+ add r3, r3705
+ add r4, r4706
+ add r5, r5707
+ lea r6, [3 * r3]708
+ lea r7, [3 * r4]709
+ lea r8, [3 * r5]710
+ PROCESS_ADDAVG_ALIGNED_16x4_HBD_AVX512711
+ RET712
+%endif713
+714
+%macro ADDAVG_ALIGNED_W16_HBD_AVX512 1715
+INIT_ZMM avx512716
+cglobal addAvg_aligned_16x%1, 6,9,6717
+ vbroadcasti32x8 m4, [pw_ %+ ADDAVG_ROUND]718
+ vbroadcasti32x8 m5, [pw_pixel_max]719
+ vbroadcasti32x8 m3, [pw_ %+ ADDAVG_FACTOR]720
+ pxor m2, m2721
+ add r3, r3722
+ add r4, r4723
+ add r5, r5724
+ lea r6, [3 * r3]725
+ lea r7, [3 * r4]726
+ lea r8, [3 * r5]727
+728
+%rep %1/4 - 1729
+ PROCESS_ADDAVG_ALIGNED_16x4_HBD_AVX512730
+ lea r2, [r2 + 4 * r5]731
+ lea r0, [r0 + 4 * r3]732
+ lea r1, [r1 + 4 * r4]733
+%endrep734
+ PROCESS_ADDAVG_ALIGNED_16x4_HBD_AVX512735
+ RET736
+%endmacro737
+738
+%if ARCH_X86_64739
+ADDAVG_ALIGNED_W16_HBD_AVX512 8740
+ADDAVG_ALIGNED_W16_HBD_AVX512 12741
+ADDAVG_ALIGNED_W16_HBD_AVX512 16742
+ADDAVG_ALIGNED_W16_HBD_AVX512 24743
+ADDAVG_ALIGNED_W16_HBD_AVX512 32744
+ADDAVG_ALIGNED_W16_HBD_AVX512 64745
+%endif746
+747
+%macro ADDAVG_ALIGNED_W32_HBD_AVX512 1748
+INIT_ZMM avx512749
+cglobal addAvg_aligned_32x%1, 6,9,6750
+ vbroadcasti32x8 m4, [pw_ %+ ADDAVG_ROUND]751
+ vbroadcasti32x8 m5, [pw_pixel_max]752
+ vbroadcasti32x8 m3, [pw_ %+ ADDAVG_FACTOR]753
+ pxor m2, m2754
+ add r3, r3755
+ add r4, r4756
+ add r5, r5757
+ lea r6, [3 * r3]758
+ lea r7, [3 * r4]759
+ lea r8, [3 * r5]760
+761
+%rep %1/4 - 1762
+ PROCESS_ADDAVG_ALIGNED_32x4_HBD_AVX512763
+ lea r2, [r2 + 4 * r5]764
+ lea r0, [r0 + 4 * r3]765
+ lea r1, [r1 + 4 * r4]766
+%endrep767
+ PROCESS_ADDAVG_ALIGNED_32x4_HBD_AVX512768
+ RET769
+%endmacro770
+771
+%if ARCH_X86_64772
+ADDAVG_ALIGNED_W32_HBD_AVX512 8773
+ADDAVG_ALIGNED_W32_HBD_AVX512 16774
+ADDAVG_ALIGNED_W32_HBD_AVX512 24775
+ADDAVG_ALIGNED_W32_HBD_AVX512 32776
+ADDAVG_ALIGNED_W32_HBD_AVX512 48777
+ADDAVG_ALIGNED_W32_HBD_AVX512 64778
+%endif779
+780
+%macro ADDAVG_ALIGNED_W64_HBD_AVX512 1781
+INIT_ZMM avx512782
+cglobal addAvg_aligned_64x%1, 6,9,6783
+ vbroadcasti32x8 m4, [pw_ %+ ADDAVG_ROUND]784
+ vbroadcasti32x8 m5, [pw_pixel_max]785
+ vbroadcasti32x8 m3, [pw_ %+ ADDAVG_FACTOR]786
+ pxor m2, m2787
+ add r3, r3788
+ add r4, r4789
+ add r5, r5790
+ lea r6, [3 * r3]791
+ lea r7, [3 * r4]792
+ lea r8, [3 * r5]793
+794
+%rep %1/4 - 1795
+ PROCESS_ADDAVG_ALIGNED_64x4_HBD_AVX512796
+ lea r2, [r2 + 4 * r5]797
+ lea r0, [r0 + 4 * r3]798
+ lea r1, [r1 + 4 * r4]799
+%endrep800
+ PROCESS_ADDAVG_ALIGNED_64x4_HBD_AVX512801
+ RET802
+%endmacro803
+804
+%if ARCH_X86_64805
+ADDAVG_ALIGNED_W64_HBD_AVX512 16806
+ADDAVG_ALIGNED_W64_HBD_AVX512 32807
+ADDAVG_ALIGNED_W64_HBD_AVX512 48808
+ADDAVG_ALIGNED_W64_HBD_AVX512 64809
+%endif810
+811
+%if ARCH_X86_64812
+INIT_ZMM avx512813
+cglobal addAvg_aligned_48x64, 6,9,6814
+ vbroadcasti32x8 m4, [pw_ %+ ADDAVG_ROUND]815
+ vbroadcasti32x8 m5, [pw_pixel_max]816
+ vbroadcasti32x8 m3, [pw_ %+ ADDAVG_FACTOR]817
+ pxor m2, m2818
+ add r3, r3819
+ add r4, r4820
+ add r5, r5821
+ lea r6, [3 * r3]822
+ lea r7, [3 * r4]823
+ lea r8, [3 * r5]824
+825
+%rep 15826
+ PROCESS_ADDAVG_ALIGNED_48x4_HBD_AVX512827
+ lea r2, [r2 + 4 * r5]828
+ lea r0, [r0 + 4 * r3]829
+ lea r1, [r1 + 4 * r4]830
+%endrep831
+ PROCESS_ADDAVG_ALIGNED_48x4_HBD_AVX512832
+ RET833
+%endif834
+;-----------------------------------------------------------------------------835
+;addAvg avx512 high bit depth code end836
+;-----------------------------------------------------------------------------837
;-----------------------------------------------------------------------------838
%else ; !HIGH_BIT_DEPTH839
;-----------------------------------------------------------------------------840
841
;-----------------------------------------------------------------------------842
; addAvg avx2 code end843
;-----------------------------------------------------------------------------844
+; addAvg avx512 code start845
+;-----------------------------------------------------------------------------846
+%macro PROCESS_ADDAVG_64x2_AVX512 0847
+ movu m0, [r0]848
+ movu m1, [r1]849
+ movu m2, [r0 + mmsize]850
+ movu m3, [r1 + mmsize]851
+852
+ paddw m0, m1853
+ pmulhrsw m0, m4854
+ paddw m0, m5855
+ paddw m2, m3856
+ pmulhrsw m2, m4857
+ paddw m2, m5858
859
+ packuswb m0, m2860
+ vpermq m0, m6, m0861
+ movu [r2], m0862
+863
+ movu m0, [r0 + r3]864
+ movu m1, [r1 + r4]865
+ movu m2, [r0 + r3 + mmsize]866
+ movu m3, [r1 + r4 + mmsize]867
+868
+ paddw m0, m1869
+ pmulhrsw m0, m4870
+ paddw m0, m5871
+ paddw m2, m3872
+ pmulhrsw m2, m4873
+ paddw m2, m5874
+875
+ packuswb m0, m2876
+ vpermq m0, m6, m0877
+ movu [r2 + r5], m0878
+%endmacro879
+880
+%macro PROCESS_ADDAVG_32x2_AVX512 0881
+ movu m0, [r0]882
+ movu m1, [r1]883
+ movu m2, [r0 + r3]884
+ movu m3, [r1 + r4]885
+886
+ paddw m0, m1887
+ pmulhrsw m0, m4888
+ paddw m0, m5889
+ paddw m2, m3890
+ pmulhrsw m2, m4891
+ paddw m2, m5892
+893
+ packuswb m0, m2894
+ vpermq m0, m6, m0895
+ movu [r2], ym0896
+ vextracti32x8 [r2 + r5], m0, 1897
+%endmacro898
+;--------------------------------------------------------------------------------------------------------------------899
+;void addAvg (int16_t* src0, int16_t* src1, pixel* dst, intptr_t src0Stride, intptr_t src1Stride, intptr_t dstStride)900
+;--------------------------------------------------------------------------------------------------------------------901
+%macro ADDAVG_W64_AVX512 1902
+INIT_ZMM avx512903
+cglobal addAvg_64x%1, 6,6,7904
+ vbroadcasti32x8 m4, [pw_256]905
+ vbroadcasti32x8 m5, [pw_128]906
+ mova m6, [shuf_avx512]907
+908
+ add r3, r3909
+ add r4, r4910
+911
+%rep %1/2 - 1912
+ PROCESS_ADDAVG_64x2_AVX512913
+ lea r2, [r2 + 2 * r5]914
+ lea r0, [r0 + 2 * r3]915
+ lea r1, [r1 + 2 * r4]916
+%endrep917
+ PROCESS_ADDAVG_64x2_AVX512918
+ RET919
+%endmacro920
+921
+ADDAVG_W64_AVX512 16922
+ADDAVG_W64_AVX512 32923
+ADDAVG_W64_AVX512 48924
+ADDAVG_W64_AVX512 64925
+926
+%macro ADDAVG_W32_AVX512 1927
+INIT_ZMM avx512928
+cglobal addAvg_32x%1, 6,6,7929
+ vbroadcasti32x8 m4, [pw_256]930
+ vbroadcasti32x8 m5, [pw_128]931
+ mova m6, [shuf_avx512]932
+ add r3, r3933
+ add r4, r4934
+935
+%rep %1/2 - 1936
+ PROCESS_ADDAVG_32x2_AVX512937
+ lea r2, [r2 + 2 * r5]938
+ lea r0, [r0 + 2 * r3]939
+ lea r1, [r1 + 2 * r4]940
+%endrep941
+ PROCESS_ADDAVG_32x2_AVX512942
+ RET943
+%endmacro944
+945
+ADDAVG_W32_AVX512 8946
+ADDAVG_W32_AVX512 16947
+ADDAVG_W32_AVX512 24948
+ADDAVG_W32_AVX512 32949
+ADDAVG_W32_AVX512 48950
+ADDAVG_W32_AVX512 64951
+952
+%macro PROCESS_ADDAVG_ALIGNED_64x2_AVX512 0953
+ mova m0, [r0]954
+ mova m1, [r1]955
+ mova m2, [r0 + mmsize]956
+ mova m3, [r1 + mmsize]957
+958
+ paddw m0, m1959
+ pmulhrsw m0, m4960
+ paddw m0, m5961
+ paddw m2, m3962
+ pmulhrsw m2, m4963
+ paddw m2, m5964
+965
+ packuswb m0, m2966
+ vpermq m0, m6, m0967
+ mova [r2], m0968
+969
+ mova m0, [r0 + r3]970
+ mova m1, [r1 + r4]971
+ mova m2, [r0 + r3 + mmsize]972
+ mova m3, [r1 + r4 + mmsize]973
+974
+ paddw m0, m1975
+ pmulhrsw m0, m4976
+ paddw m0, m5977
+ paddw m2, m3978
+ pmulhrsw m2, m4979
+ paddw m2, m5980
+981
+ packuswb m0, m2982
+ vpermq m0, m6, m0983
+ mova [r2 + r5], m0984
+%endmacro985
+986
+%macro PROCESS_ADDAVG_ALIGNED_32x2_AVX512 0987
+ mova m0, [r0]988
+ mova m1, [r1]989
+ mova m2, [r0 + r3]990
+ mova m3, [r1 + r4]991
+992
+ paddw m0, m1993
+ pmulhrsw m0, m4994
+ paddw m0, m5995
+ paddw m2, m3996
+ pmulhrsw m2, m4997
+ paddw m2, m5998
+999
+ packuswb m0, m21000
+ vpermq m0, m6, m01001
+ mova [r2], ym01002
+ vextracti32x8 [r2 + r5], m0, 11003
+%endmacro1004
+;--------------------------------------------------------------------------------------------------------------------1005
+;void addAvg (int16_t* src0, int16_t* src1, pixel* dst, intptr_t src0Stride, intptr_t src1Stride, intptr_t dstStride)1006
+;--------------------------------------------------------------------------------------------------------------------1007
+%macro ADDAVG_ALIGNED_W64_AVX512 11008
+INIT_ZMM avx5121009
+cglobal addAvg_aligned_64x%1, 6,6,71010
+ vbroadcasti32x8 m4, [pw_256]1011
+ vbroadcasti32x8 m5, [pw_128]1012
+ mova m6, [shuf_avx512]1013
+1014
+ add r3, r31015
+ add r4, r41016
+1017
+%rep %1/2 - 11018
+ PROCESS_ADDAVG_ALIGNED_64x2_AVX5121019
+ lea r2, [r2 + 2 * r5]1020
+ lea r0, [r0 + 2 * r3]1021
+ lea r1, [r1 + 2 * r4]1022
+%endrep1023
+ PROCESS_ADDAVG_ALIGNED_64x2_AVX5121024
+ RET1025
+%endmacro1026
+1027
+ADDAVG_ALIGNED_W64_AVX512 161028
+ADDAVG_ALIGNED_W64_AVX512 321029
+ADDAVG_ALIGNED_W64_AVX512 481030
+ADDAVG_ALIGNED_W64_AVX512 641031
+1032
+%macro ADDAVG_ALIGNED_W32_AVX512 11033
+INIT_ZMM avx5121034
+cglobal addAvg_aligned_32x%1, 6,6,71035
+ vbroadcasti32x8 m4, [pw_256]1036
+ vbroadcasti32x8 m5, [pw_128]1037
+ mova m6, [shuf_avx512]1038
+ add r3, r31039
+ add r4, r41040
+1041
+%rep %1/2 - 11042
+ PROCESS_ADDAVG_ALIGNED_32x2_AVX5121043
+ lea r2, [r2 + 2 * r5]1044
+ lea r0, [r0 + 2 * r3]1045
+ lea r1, [r1 + 2 * r4]1046
+%endrep1047
+ PROCESS_ADDAVG_ALIGNED_32x2_AVX5121048
+ RET1049
+%endmacro1050
+1051
+ADDAVG_ALIGNED_W32_AVX512 81052
+ADDAVG_ALIGNED_W32_AVX512 161053
+ADDAVG_ALIGNED_W32_AVX512 241054
+ADDAVG_ALIGNED_W32_AVX512 321055
+ADDAVG_ALIGNED_W32_AVX512 481056
+ADDAVG_ALIGNED_W32_AVX512 641057
+;-----------------------------------------------------------------------------1058
+; addAvg avx512 code end1059
;-----------------------------------------------------------------------------1060
%macro ADDAVG_W24_H2 21061
INIT_XMM sse41062
1063
%endmacro1064
%endif1065
1066
-%macro AVG_END 01067
- lea t4, [t4+t5*2*SIZEOF_PIXEL]1068
+%macro AVG_END 0-1 2;rows1069
lea t2, [t2+t3*2*SIZEOF_PIXEL]1070
+ lea t4, [t4+t5*2*SIZEOF_PIXEL]1071
lea t0, [t0+t1*2*SIZEOF_PIXEL]1072
- sub eax, 21073
+ sub eax, %11074
jg .height_loop1075
%ifidn movu,movq ; detect MMX1076
EMMS1077
1078
%endmacro1079
1080
%macro BIWEIGHT_START_SSSE3 01081
- movzx t6d, byte r6m ; FIXME x86_641082
- mov t7d, 641083
- sub t7d, t6d1084
- shl t7d, 81085
- add t6d, t7d1086
- mova m4, [pw_512]1087
- movd xm3, t6d1088
+ movzx t6d, byte r6m ; FIXME x86_641089
+%if mmsize > 161090
+ vbroadcasti128 m4, [pw_512]1091
+%else1092
+ mova m4, [pw_512]1093
+%endif1094
+ lea t7d, [t6+(64<<8)]1095
+ shl t6d, 81096
+ sub t7d, t6d1097
+%if cpuflag(avx512)1098
+ vpbroadcastw m3, t7d1099
+%else1100
+ movd xm3, t7d1101
%if cpuflag(avx2)1102
- vpbroadcastw m3, xm31103
+ vpbroadcastw m3, xm31104
%else1105
- SPLATW m3, m3 ; weight_dst,src1106
+ SPLATW m3, m3 ; weight_dst,src1107
+%endif1108
%endif1109
%endmacro1110
1111
1112
AVG_WEIGHT 24, 71113
AVG_WEIGHT 48, 71114
1115
+INIT_YMM avx5121116
+cglobal pixel_avg_weight_w81117
+ BIWEIGHT_START1118
+ kxnorb k1, k1, k11119
+ kaddb k1, k1, k11120
+ AVG_START 51121
+.height_loop:1122
+ movq xm0, [t2]1123
+ movq xm2, [t4]1124
+ movq xm1, [t2+t3]1125
+ movq xm5, [t4+t5]1126
+ lea t2, [t2+t3*2]1127
+ lea t4, [t4+t5*2]1128
+ vpbroadcastq m0 {k1}, [t2]1129
+ vpbroadcastq m2 {k1}, [t4]1130
+ vpbroadcastq m1 {k1}, [t2+t3]1131
+ vpbroadcastq m5 {k1}, [t4+t5]1132
+ punpcklbw m0, m21133
+ punpcklbw m1, m51134
+ pmaddubsw m0, m31135
+ pmaddubsw m1, m31136
+ pmulhrsw m0, m41137
+ pmulhrsw m1, m41138
+ packuswb m0, m11139
+ vextracti128 xmm1, m0, 11140
+ movq [t0], xm01141
+ movhps [t0+t1], xm01142
+ lea t0, [t0+t1*2]1143
+ movq [t0], xmm11144
+ movhps [t0+t1], xmm11145
+ AVG_END 41146
+1147
INIT_YMM avx21148
cglobal pixel_avg_weight_w161149
BIWEIGHT_START1150
1151
vextracti128 [t0+t1], m0, 11152
AVG_END1153
1154
+INIT_ZMM avx5121155
+ cglobal pixel_avg_weight_w161156
+ BIWEIGHT_START1157
+ AVG_START 51158
+.height_loop:1159
+ movu xm0, [t2]1160
+ movu xm1, [t4]1161
+ vinserti128 ym0, [t2+t3], 11162
+ vinserti128 ym1, [t4+t5], 11163
+ lea t2, [t2+t3*2]1164
+ lea t4, [t4+t5*2]1165
+ vinserti32x4 m0, [t2], 21166
+ vinserti32x4 m1, [t4], 21167
+ vinserti32x4 m0, [t2+t3], 31168
+ vinserti32x4 m1, [t4+t5], 31169
+ SBUTTERFLY bw, 0, 1, 21170
+ pmaddubsw m0, m31171
+ pmaddubsw m1, m31172
+ pmulhrsw m0, m41173
+ pmulhrsw m1, m41174
+ packuswb m0, m11175
+ mova [t0], xm01176
+ vextracti128 [t0+t1], ym0, 11177
+ lea t0, [t0+t1*2]1178
+ vextracti32x4 [t0], m0, 21179
+ vextracti32x4 [t0+t1], m0, 31180
+ AVG_END 41181
+1182
+INIT_YMM avx21183
cglobal pixel_avg_weight_w321184
BIWEIGHT_START1185
AVG_START 51186
1187
mova [t0], m01188
AVG_END1189
1190
+INIT_YMM avx21191
cglobal pixel_avg_weight_w641192
BIWEIGHT_START1193
AVG_START 51194
1195
AVGH 16, 81196
AVGH 16, 41197
1198
+INIT_XMM avx5121199
+AVGH 16, 641200
+AVGH 16, 321201
+AVGH 16, 161202
+AVGH 16, 121203
+AVGH 16, 81204
+AVGH 16, 41205
+AVGH 8, 321206
+AVGH 8, 161207
+AVGH 8, 81208
+AVGH 8, 41209
+1210
%endif ;HIGH_BIT_DEPTH1211
1212
;-------------------------------------------------------------------------------------------------------------------------------1213
1214
RET1215
%endif1216
1217
+;-----------------------------------------------------------------------------1218
+;pixel_avg_pp avx512 code start1219
+;-----------------------------------------------------------------------------1220
+%macro PROCESS_PIXELAVG_64x4_AVX512 01221
+ movu m0, [r2]1222
+ movu m2, [r2 + r3]1223
+ movu m1, [r4]1224
+ movu m3, [r4 + r5]1225
+ pavgb m0, m11226
+ pavgb m2, m31227
+ movu [r0], m01228
+ movu [r0 + r1], m21229
+1230
+ movu m0, [r2 + 2 * r3]1231
+ movu m2, [r2 + r7]1232
+ movu m1, [r4 + 2 * r5]1233
+ movu m3, [r4 + r8]1234
+ pavgb m0, m11235
+ pavgb m2, m31236
+ movu [r0 + 2 * r1], m01237
+ movu [r0 + r6], m21238
+%endmacro1239
+1240
+;-------------------------------------------------------------------------------------------------------------------------------1241
+;void pixelavg_pp(pixel* dst, intptr_t dstride, const pixel* src0, intptr_t sstride0, const pixel* src1, intptr_t sstride1, int)1242
+;-------------------------------------------------------------------------------------------------------------------------------1243
+%if ARCH_X86_64 && BIT_DEPTH == 81244
+%macro PIXEL_AVG_64xN_AVX512 11245
+INIT_ZMM avx5121246
+cglobal pixel_avg_64x%1, 6, 9, 41247
+ lea r6, [3 * r1]1248
+ lea r7, [3 * r3]1249
+ lea r8, [3 * r5]1250
+1251
+%rep %1/4 - 11252
+ PROCESS_PIXELAVG_64x4_AVX5121253
+ lea r2, [r2 + r3 * 4]1254
+ lea r4, [r4 + r5 * 4]1255
+ lea r0, [r0 + r1 * 4]1256
+%endrep1257
+ PROCESS_PIXELAVG_64x4_AVX5121258
+ RET1259
+%endmacro1260
+1261
+PIXEL_AVG_64xN_AVX512 161262
+PIXEL_AVG_64xN_AVX512 321263
+PIXEL_AVG_64xN_AVX512 481264
+PIXEL_AVG_64xN_AVX512 641265
+%endif1266
+;-----------------------------------------------------------------------------1267
+;pixel_avg_pp avx512 code end1268
+;-----------------------------------------------------------------------------1269
;=============================================================================1270
; pixel avg21271
;=============================================================================1272
1273
RET1274
%endif1275
1276
+;-----------------------------------------------------------------------------1277
+;pixel_avg_pp avx512 high bit depth code start1278
+;-----------------------------------------------------------------------------1279
+%macro PROCESS_PIXELAVG_32x8_HBD_AVX512 01280
+ movu m0, [r2]1281
+ movu m1, [r4]1282
+ movu m2, [r2 + r3]1283
+ movu m3, [r4 + r5]1284
+ pavgw m0, m11285
+ pavgw m2, m31286
+ movu [r0], m01287
+ movu [r0 + r1], m21288
+1289
+ movu m0, [r2 + r3 * 2]1290
+ movu m1, [r4 + r5 * 2]1291
+ movu m2, [r2 + r6]1292
+ movu m3, [r4 + r7]1293
+ pavgw m0, m11294
+ pavgw m2, m31295
+ movu [r0 + r1 * 2], m01296
+ movu [r0 + r8], m21297
+1298
+ lea r0, [r0 + 4 * r1]1299
+ lea r2, [r2 + 4 * r3]1300
+ lea r4, [r4 + 4 * r5]1301
+1302
+ movu m0, [r2]1303
+ movu m1, [r4]1304
+ movu m2, [r2 + r3]1305
+ movu m3, [r4 + r5]1306
+ pavgw m0, m11307
+ pavgw m2, m31308
+ movu [r0], m01309
+ movu [r0 + r1], m21310
+1311
+ movu m0, [r2 + r3 * 2]1312
+ movu m1, [r4 + r5 * 2]1313
+ movu m2, [r2 + r6]1314
+ movu m3, [r4 + r7]1315
+ pavgw m0, m11316
+ pavgw m2, m31317
+ movu [r0 + r1 * 2], m01318
+ movu [r0 + r8], m21319
+%endmacro1320
+%macro PROCESS_PIXELAVG_ALIGNED_32x8_HBD_AVX512 01321
+ mova m0, [r2]1322
+ mova m1, [r4]1323
+ mova m2, [r2 + r3]1324
+ mova m3, [r4 + r5]1325
+ pavgw m0, m11326
+ pavgw m2, m31327
+ mova [r0], m01328
+ mova [r0 + r1], m21329
+1330
+ mova m0, [r2 + r3 * 2]1331
+ mova m1, [r4 + r5 * 2]1332
+ mova m2, [r2 + r6]1333
+ mova m3, [r4 + r7]1334
+ pavgw m0, m11335
+ pavgw m2, m31336
+ mova [r0 + r1 * 2], m01337
+ mova [r0 + r8], m21338
+1339
+ lea r0, [r0 + 4 * r1]1340
+ lea r2, [r2 + 4 * r3]1341
+ lea r4, [r4 + 4 * r5]1342
+1343
+ mova m0, [r2]1344
+ mova m1, [r4]1345
+ mova m2, [r2 + r3]1346
+ mova m3, [r4 + r5]1347
+ pavgw m0, m11348
+ pavgw m2, m31349
+ mova [r0], m01350
+ mova [r0 + r1], m21351
+1352
+ mova m0, [r2 + r3 * 2]1353
+ mova m1, [r4 + r5 * 2]1354
+ mova m2, [r2 + r6]1355
+ mova m3, [r4 + r7]1356
+ pavgw m0, m11357
+ pavgw m2, m31358
+ mova [r0 + r1 * 2], m01359
+ mova [r0 + r8], m21360
+%endmacro1361
+1362
+%macro PROCESS_PIXELAVG_64x8_HBD_AVX512 01363
+ movu m0, [r2]1364
+ movu m1, [r4]1365
+ movu m2, [r2 + r3]1366
+ movu m3, [r4 + r5]1367
+ pavgw m0, m11368
+ pavgw m2, m31369
+ movu [r0], m01370
+ movu [r0 + r1], m21371
+1372
+ movu m0, [r2 + mmsize]1373
+ movu m1, [r4 + mmsize]1374
+ movu m2, [r2 + r3 + mmsize]1375
+ movu m3, [r4 + r5 + mmsize]1376
+ pavgw m0, m11377
+ pavgw m2, m31378
+ movu [r0 + mmsize], m01379
+ movu [r0 + r1 + mmsize], m21380
+1381
+ movu m0, [r2 + r3 * 2]1382
+ movu m1, [r4 + r5 * 2]1383
+ movu m2, [r2 + r6]1384
+ movu m3, [r4 + r7]1385
+ pavgw m0, m11386
+ pavgw m2, m31387
+ movu [r0 + r1 * 2], m01388
+ movu [r0 + r8], m21389
+1390
+ movu m0, [r2 + r3 * 2 + mmsize]1391
+ movu m1, [r4 + r5 * 2 + mmsize]1392
+ movu m2, [r2 + r6 + mmsize]1393
+ movu m3, [r4 + r7 + mmsize]1394
+ pavgw m0, m11395
+ pavgw m2, m31396
+ movu [r0 + r1 * 2 + mmsize], m01397
+ movu [r0 + r8 + mmsize], m21398
+1399
+ lea r0, [r0 + 4 * r1]1400
+ lea r2, [r2 + 4 * r3]1401
+ lea r4, [r4 + 4 * r5]1402
+1403
+ movu m0, [r2]1404
+ movu m1, [r4]1405
+ movu m2, [r2 + r3]1406
+ movu m3, [r4 + r5]1407
+ pavgw m0, m11408
+ pavgw m2, m31409
+ movu [r0], m01410
+ movu [r0 + r1], m21411
+1412
+ movu m0, [r2 + mmsize]1413
+ movu m1, [r4 + mmsize]1414
+ movu m2, [r2 + r3 + mmsize]1415
+ movu m3, [r4 + r5 + mmsize]1416
+ pavgw m0, m11417
+ pavgw m2, m31418
+ movu [r0 + mmsize], m01419
+ movu [r0 + r1 + mmsize], m21420
+1421
+ movu m0, [r2 + r3 * 2]1422
+ movu m1, [r4 + r5 * 2]1423
+ movu m2, [r2 + r6]1424
+ movu m3, [r4 + r7]1425
+ pavgw m0, m11426
+ pavgw m2, m31427
+ movu [r0 + r1 * 2], m01428
+ movu [r0 + r8], m21429
+1430
+ movu m0, [r2 + r3 * 2 + mmsize]1431
+ movu m1, [r4 + r5 * 2 + mmsize]1432
+ movu m2, [r2 + r6 + mmsize]1433
+ movu m3, [r4 + r7 + mmsize]1434
+ pavgw m0, m11435
+ pavgw m2, m31436
+ movu [r0 + r1 * 2 + mmsize], m01437
+ movu [r0 + r8 + mmsize], m21438
+%endmacro1439
+%macro PROCESS_PIXELAVG_ALIGNED_64x8_HBD_AVX512 01440
+ mova m0, [r2]1441
+ mova m1, [r4]1442
+ mova m2, [r2 + r3]1443
+ mova m3, [r4 + r5]1444
+ pavgw m0, m11445
+ pavgw m2, m31446
+ mova [r0], m01447
+ mova [r0 + r1], m21448
+1449
+ mova m0, [r2 + mmsize]1450
+ mova m1, [r4 + mmsize]1451
+ mova m2, [r2 + r3 + mmsize]1452
+ mova m3, [r4 + r5 + mmsize]1453
+ pavgw m0, m11454
+ pavgw m2, m31455
+ mova [r0 + mmsize], m01456
+ mova [r0 + r1 + mmsize], m21457
+1458
+ mova m0, [r2 + r3 * 2]1459
+ mova m1, [r4 + r5 * 2]1460
+ mova m2, [r2 + r6]1461
+ mova m3, [r4 + r7]1462
+ pavgw m0, m11463
+ pavgw m2, m31464
+ mova [r0 + r1 * 2], m01465
+ mova [r0 + r8], m21466
+1467
+ mova m0, [r2 + r3 * 2 + mmsize]1468
+ mova m1, [r4 + r5 * 2 + mmsize]1469
+ mova m2, [r2 + r6 + mmsize]1470
+ mova m3, [r4 + r7 + mmsize]1471
+ pavgw m0, m11472
+ pavgw m2, m31473
+ mova [r0 + r1 * 2 + mmsize], m01474
+ mova [r0 + r8 + mmsize], m21475
+1476
+ lea r0, [r0 + 4 * r1]1477
+ lea r2, [r2 + 4 * r3]1478
+ lea r4, [r4 + 4 * r5]1479
+1480
+ mova m0, [r2]1481
+ mova m1, [r4]1482
+ mova m2, [r2 + r3]1483
+ mova m3, [r4 + r5]1484
+ pavgw m0, m11485
+ pavgw m2, m31486
+ mova [r0], m01487
+ mova [r0 + r1], m21488
+1489
+ mova m0, [r2 + mmsize]1490
+ mova m1, [r4 + mmsize]1491
+ mova m2, [r2 + r3 + mmsize]1492
+ mova m3, [r4 + r5 + mmsize]1493
+ pavgw m0, m11494
+ pavgw m2, m31495
+ mova [r0 + mmsize], m01496
+ mova [r0 + r1 + mmsize], m21497
+1498
+ mova m0, [r2 + r3 * 2]1499
+ mova m1, [r4 + r5 * 2]1500
+ mova m2, [r2 + r6]1501
+ mova m3, [r4 + r7]1502
+ pavgw m0, m11503
+ pavgw m2, m31504
+ mova [r0 + r1 * 2], m01505
+ mova [r0 + r8], m21506
+1507
+ mova m0, [r2 + r3 * 2 + mmsize]1508
+ mova m1, [r4 + r5 * 2 + mmsize]1509
+ mova m2, [r2 + r6 + mmsize]1510
+ mova m3, [r4 + r7 + mmsize]1511
+ pavgw m0, m11512
+ pavgw m2, m31513
+ mova [r0 + r1 * 2 + mmsize], m01514
+ mova [r0 + r8 + mmsize], m21515
+%endmacro1516
+1517
+%macro PROCESS_PIXELAVG_48x8_HBD_AVX512 01518
+ movu m0, [r2]1519
+ movu m1, [r4]1520
+ movu m2, [r2 + r3]1521
+ movu m3, [r4 + r5]1522
+ pavgw m0, m11523
+ pavgw m2, m31524
+ movu [r0], m01525
+ movu [r0 + r1], m21526
+1527
+ movu ym0, [r2 + mmsize]1528
+ movu ym1, [r4 + mmsize]1529
+ movu ym2, [r2 + r3 + mmsize]1530
+ movu ym3, [r4 + r5 + mmsize]1531
+ pavgw ym0, ym11532
+ pavgw ym2, ym31533
+ movu [r0 + mmsize], ym01534
+ movu [r0 + r1 + mmsize], ym21535
+1536
+ movu m0, [r2 + r3 * 2]1537
+ movu m1, [r4 + r5 * 2]1538
+ movu m2, [r2 + r6]1539
+ movu m3, [r4 + r7]1540
+ pavgw m0, m11541
+ pavgw m2, m31542
+ movu [r0 + r1 * 2], m01543
+ movu [r0 + r8], m21544
+1545
+ movu ym0, [r2 + r3 * 2 + mmsize]1546
+ movu ym1, [r4 + r5 * 2 + mmsize]1547
+ movu ym2, [r2 + r6 + mmsize]1548
+ movu ym3, [r4 + r7 + mmsize]1549
+ pavgw ym0, ym11550
+ pavgw ym2, ym31551
+ movu [r0 + r1 * 2 + mmsize], ym01552
+ movu [r0 + r8 + mmsize], ym21553
+1554
+ lea r0, [r0 + 4 * r1]1555
+ lea r2, [r2 + 4 * r3]1556
+ lea r4, [r4 + 4 * r5]1557
+1558
+ movu m0, [r2]1559
+ movu m1, [r4]1560
+ movu m2, [r2 + r3]1561
+ movu m3, [r4 + r5]1562
+ pavgw m0, m11563
+ pavgw m2, m31564
+ movu [r0], m01565
+ movu [r0 + r1], m21566
+1567
+ movu ym0, [r2 + mmsize]1568
+ movu ym1, [r4 + mmsize]1569
+ movu ym2, [r2 + r3 + mmsize]1570
+ movu ym3, [r4 + r5 + mmsize]1571
+ pavgw ym0, ym11572
+ pavgw ym2, ym31573
+ movu [r0 + mmsize], ym01574
+ movu [r0 + r1 + mmsize], ym21575
+1576
+ movu m0, [r2 + r3 * 2]1577
+ movu m1, [r4 + r5 * 2]1578
+ movu m2, [r2 + r6]1579
+ movu m3, [r4 + r7]1580
+ pavgw m0, m11581
+ pavgw m2, m31582
+ movu [r0 + r1 * 2], m01583
+ movu [r0 + r8], m21584
+1585
+ movu ym0, [r2 + r3 * 2 + mmsize]1586
+ movu ym1, [r4 + r5 * 2 + mmsize]1587
+ movu ym2, [r2 + r6 + mmsize]1588
+ movu ym3, [r4 + r7 + mmsize]1589
+ pavgw ym0, ym11590
+ pavgw ym2, ym31591
+ movu [r0 + r1 * 2 + mmsize], ym01592
+ movu [r0 + r8 + mmsize], ym21593
+%endmacro1594
+%macro PROCESS_PIXELAVG_ALIGNED_48x8_HBD_AVX512 01595
+ mova m0, [r2]1596
+ mova m1, [r4]1597
+ mova m2, [r2 + r3]1598
+ mova m3, [r4 + r5]1599
+ pavgw m0, m11600
+ pavgw m2, m31601
+ mova [r0], m01602
+ mova [r0 + r1], m21603
+1604
+ mova ym0, [r2 + mmsize]1605
+ mova ym1, [r4 + mmsize]1606
+ mova ym2, [r2 + r3 + mmsize]1607
+ mova ym3, [r4 + r5 + mmsize]1608
+ pavgw ym0, ym11609
+ pavgw ym2, ym31610
+ mova [r0 + mmsize], ym01611
+ mova [r0 + r1 + mmsize], ym21612
+1613
+ mova m0, [r2 + r3 * 2]1614
+ mova m1, [r4 + r5 * 2]1615
+ mova m2, [r2 + r6]1616
+ mova m3, [r4 + r7]1617
+ pavgw m0, m11618
+ pavgw m2, m31619
+ mova [r0 + r1 * 2], m01620
+ mova [r0 + r8], m21621
+1622
+ mova ym0, [r2 + r3 * 2 + mmsize]1623
+ mova ym1, [r4 + r5 * 2 + mmsize]1624
+ mova ym2, [r2 + r6 + mmsize]1625
+ mova ym3, [r4 + r7 + mmsize]1626
+ pavgw ym0, ym11627
+ pavgw ym2, ym31628
+ mova [r0 + r1 * 2 + mmsize], ym01629
+ mova [r0 + r8 + mmsize], ym21630
+1631
+ lea r0, [r0 + 4 * r1]1632
+ lea r2, [r2 + 4 * r3]1633
+ lea r4, [r4 + 4 * r5]1634
+1635
+ mova m0, [r2]1636
+ mova m1, [r4]1637
+ mova m2, [r2 + r3]1638
+ mova m3, [r4 + r5]1639
+ pavgw m0, m11640
+ pavgw m2, m31641
+ mova [r0], m01642
+ mova [r0 + r1], m21643
+1644
+ mova ym0, [r2 + mmsize]1645
+ mova ym1, [r4 + mmsize]1646
+ mova ym2, [r2 + r3 + mmsize]1647
+ mova ym3, [r4 + r5 + mmsize]1648
+ pavgw ym0, ym11649
+ pavgw ym2, ym31650
+ mova [r0 + mmsize], ym01651
+ mova [r0 + r1 + mmsize], ym21652
+1653
+ mova m0, [r2 + r3 * 2]1654
+ mova m1, [r4 + r5 * 2]1655
+ mova m2, [r2 + r6]1656
+ mova m3, [r4 + r7]1657
+ pavgw m0, m11658
+ pavgw m2, m31659
+ mova [r0 + r1 * 2], m01660
+ mova [r0 + r8], m21661
+1662
+ mova ym0, [r2 + r3 * 2 + mmsize]1663
+ mova ym1, [r4 + r5 * 2 + mmsize]1664
+ mova ym2, [r2 + r6 + mmsize]1665
+ mova ym3, [r4 + r7 + mmsize]1666
+ pavgw ym0, ym11667
+ pavgw ym2, ym31668
+ mova [r0 + r1 * 2 + mmsize], ym01669
+ mova [r0 + r8 + mmsize], ym21670
+%endmacro1671
+1672
+%macro PIXEL_AVG_HBD_W32 11673
+INIT_ZMM avx5121674
+cglobal pixel_avg_32x%1, 6,9,41675
+ shl r1d, 11676
+ shl r3d, 11677
+ shl r5d, 11678
+ lea r6, [r3 * 3]1679
+ lea r7, [r5 * 3]1680
+ lea r8, [r1 * 3]1681
+1682
+%rep %1/8 - 11683
+ PROCESS_PIXELAVG_32x8_HBD_AVX5121684
+ lea r0, [r0 + 4 * r1]1685
+ lea r2, [r2 + 4 * r3]1686
+ lea r4, [r4 + 4 * r5]1687
+%endrep1688
+ PROCESS_PIXELAVG_32x8_HBD_AVX5121689
+ RET1690
+%endmacro1691
+1692
+%if ARCH_X86_641693
+PIXEL_AVG_HBD_W32 81694
+PIXEL_AVG_HBD_W32 161695
+PIXEL_AVG_HBD_W32 241696
+PIXEL_AVG_HBD_W32 321697
+PIXEL_AVG_HBD_W32 641698
+%endif1699
+%macro PIXEL_AVG_HBD_ALIGNED_W32 11700
+INIT_ZMM avx5121701
+cglobal pixel_avg_aligned_32x%1, 6,9,41702
+ shl r1d, 11703
+ shl r3d, 11704
+ shl r5d, 11705
+ lea r6, [r3 * 3]1706
+ lea r7, [r5 * 3]1707
+ lea r8, [r1 * 3]1708
+1709
+%rep %1/8 - 11710
+ PROCESS_PIXELAVG_ALIGNED_32x8_HBD_AVX5121711
+ lea r0, [r0 + 4 * r1]1712
+ lea r2, [r2 + 4 * r3]1713
+ lea r4, [r4 + 4 * r5]1714
+%endrep1715
+ PROCESS_PIXELAVG_ALIGNED_32x8_HBD_AVX5121716
+ RET1717
+%endmacro1718
+1719
+%if ARCH_X86_641720
+PIXEL_AVG_HBD_ALIGNED_W32 81721
+PIXEL_AVG_HBD_ALIGNED_W32 161722
+PIXEL_AVG_HBD_ALIGNED_W32 241723
+PIXEL_AVG_HBD_ALIGNED_W32 321724
+PIXEL_AVG_HBD_ALIGNED_W32 641725
+%endif1726
+1727
+%macro PIXEL_AVG_HBD_W64 11728
+INIT_ZMM avx5121729
+cglobal pixel_avg_64x%1, 6,9,41730
+ shl r1d, 11731
+ shl r3d, 11732
+ shl r5d, 11733
+ lea r6, [r3 * 3]1734
+ lea r7, [r5 * 3]1735
+ lea r8, [r1 * 3]1736
+1737
+%rep %1/8 - 11738
+ PROCESS_PIXELAVG_64x8_HBD_AVX5121739
+ lea r0, [r0 + 4 * r1]1740
+ lea r2, [r2 + 4 * r3]1741
+ lea r4, [r4 + 4 * r5]1742
+%endrep1743
+ PROCESS_PIXELAVG_64x8_HBD_AVX5121744
+ RET1745
+%endmacro1746
+1747
+%if ARCH_X86_641748
+PIXEL_AVG_HBD_W64 161749
+PIXEL_AVG_HBD_W64 321750
+PIXEL_AVG_HBD_W64 481751
+PIXEL_AVG_HBD_W64 641752
+%endif1753
+%macro PIXEL_AVG_HBD_ALIGNED_W64 11754
+INIT_ZMM avx5121755
+cglobal pixel_avg_aligned_64x%1, 6,9,41756
+ shl r1d, 11757
+ shl r3d, 11758
+ shl r5d, 11759
+ lea r6, [r3 * 3]1760
+ lea r7, [r5 * 3]1761
+ lea r8, [r1 * 3]1762
+1763
+%rep %1/8 - 11764
+ PROCESS_PIXELAVG_ALIGNED_64x8_HBD_AVX5121765
+ lea r0, [r0 + 4 * r1]1766
+ lea r2, [r2 + 4 * r3]1767
+ lea r4, [r4 + 4 * r5]1768
+%endrep1769
+ PROCESS_PIXELAVG_ALIGNED_64x8_HBD_AVX5121770
+ RET1771
+%endmacro1772
+1773
+%if ARCH_X86_641774
+PIXEL_AVG_HBD_ALIGNED_W64 161775
+PIXEL_AVG_HBD_ALIGNED_W64 321776
+PIXEL_AVG_HBD_ALIGNED_W64 481777
+PIXEL_AVG_HBD_ALIGNED_W64 641778
+%endif1779
+1780
+%if ARCH_X86_641781
+INIT_ZMM avx5121782
+cglobal pixel_avg_48x64, 6,9,41783
+ shl r1d, 11784
+ shl r3d, 11785
+ shl r5d, 11786
+ lea r6, [r3 * 3]1787
+ lea r7, [r5 * 3]1788
+ lea r8, [r1 * 3]1789
+1790
+%rep 71791
+ PROCESS_PIXELAVG_48x8_HBD_AVX5121792
+ lea r0, [r0 + 4 * r1]1793
+ lea r2, [r2 + 4 * r3]1794
+ lea r4, [r4 + 4 * r5]1795
+%endrep1796
+ PROCESS_PIXELAVG_48x8_HBD_AVX5121797
+ RET1798
+%endif1799
+1800
+%if ARCH_X86_641801
+INIT_ZMM avx5121802
+cglobal pixel_avg_aligned_48x64, 6,9,41803
+ shl r1d, 11804
+ shl r3d, 11805
+ shl r5d, 11806
+ lea r6, [r3 * 3]1807
+ lea r7, [r5 * 3]1808
+ lea r8, [r1 * 3]1809
+1810
+%rep 71811
+ PROCESS_PIXELAVG_ALIGNED_48x8_HBD_AVX5121812
+ lea r0, [r0 + 4 * r1]1813
+ lea r2, [r2 + 4 * r3]1814
+ lea r4, [r4 + 4 * r5]1815
+%endrep1816
+ PROCESS_PIXELAVG_ALIGNED_48x8_HBD_AVX5121817
+ RET1818
+%endif1819
+;-----------------------------------------------------------------------------1820
+;pixel_avg_pp avx512 high bit depth code end1821
+;-----------------------------------------------------------------------------1822
%endif ; HIGH_BIT_DEPTH1823
1824
%if HIGH_BIT_DEPTH == 01825
1826
jg .height_loop1827
RET1828
1829
+%if ARCH_X86_641830
INIT_YMM avx21831
cglobal pixel_avg2_w20, 6,71832
sub r2, r41833
1834
sub r5d, 21835
jg .height_loop1836
RET1837
+%endif1838
1839
; Cacheline split code for processors with high latencies for loads1840
; split over cache lines. See sad-a.asm for a more detailed explanation.1841
x265_2.7.tar.gz/source/common/x86/pixel-a.asm -> x265_2.9.tar.gz/source/common/x86/pixel-a.asm
Changed
1567
1
2
times 2 dw 1, -13
times 4 dw 14
times 2 dw 1, -15
+psy_pp_shuff1: dq 0, 1, 8, 9, 4, 5, 12, 136
+psy_pp_shuff2: dq 2, 3, 10, 11, 6, 7, 14, 157
+psy_pp_shuff3: dq 0, 0, 8, 8, 1, 1, 9, 98
9
ALIGN 3210
transd_shuf1: SHUFFLE_MASK_W 0, 8, 2, 10, 4, 12, 6, 1411
12
%endif ; ARCH_X86_64=113
%endif ; HIGH_BIT_DEPTH14
15
+%macro SATD_AVX512_LOAD4 2 ; size, opmask16
+ vpbroadcast%1 m0, [r0]17
+ vpbroadcast%1 m0 {%2}, [r0+2*r1]18
+ vpbroadcast%1 m2, [r2]19
+ vpbroadcast%1 m2 {%2}, [r2+2*r3]20
+ add r0, r121
+ add r2, r322
+ vpbroadcast%1 m1, [r0]23
+ vpbroadcast%1 m1 {%2}, [r0+2*r1]24
+ vpbroadcast%1 m3, [r2]25
+ vpbroadcast%1 m3 {%2}, [r2+2*r3]26
+%endmacro27
+28
+%macro SATD_AVX512_LOAD8 5 ; size, halfreg, opmask1, opmask2, opmask329
+ vpbroadcast%1 %{2}0, [r0]30
+ vpbroadcast%1 %{2}0 {%3}, [r0+2*r1]31
+ vpbroadcast%1 %{2}2, [r2]32
+ vpbroadcast%1 %{2}2 {%3}, [r2+2*r3]33
+ vpbroadcast%1 m0 {%4}, [r0+4*r1]34
+ vpbroadcast%1 m2 {%4}, [r2+4*r3]35
+ vpbroadcast%1 m0 {%5}, [r0+2*r4]36
+ vpbroadcast%1 m2 {%5}, [r2+2*r5]37
+ vpbroadcast%1 %{2}1, [r0+r1]38
+ vpbroadcast%1 %{2}1 {%3}, [r0+r4]39
+ vpbroadcast%1 %{2}3, [r2+r3]40
+ vpbroadcast%1 %{2}3 {%3}, [r2+r5]41
+ lea r0, [r0+4*r1]42
+ lea r2, [r2+4*r3]43
+ vpbroadcast%1 m1 {%4}, [r0+r1]44
+ vpbroadcast%1 m3 {%4}, [r2+r3]45
+ vpbroadcast%1 m1 {%5}, [r0+r4]46
+ vpbroadcast%1 m3 {%5}, [r2+r5]47
+%endmacro48
+49
+%macro SATD_AVX512_PACKED 050
+ DIFF_SUMSUB_SSSE3 0, 2, 1, 3, 451
+ SUMSUB_BA w, 0, 1, 252
+ SBUTTERFLY qdq, 0, 1, 253
+ SUMSUB_BA w, 0, 1, 254
+ HMAXABSW2 0, 1, 2, 355
+%endmacro56
+57
+%macro SATD_AVX512_END 0-1 0 ; sa8d58
+ paddw m0 {k1}{z}, m1 ; zero-extend to dwords59
+%if ARCH_X86_6460
+%if mmsize == 6461
+ vextracti32x8 ym1, m0, 162
+ paddd ym0, ym163
+%endif64
+%if mmsize >= 3265
+ vextracti128 xm1, ym0, 166
+ paddd xmm0, xm0, xm167
+%endif68
+ punpckhqdq xmm1, xmm0, xmm069
+ paddd xmm0, xmm170
+ movq rax, xmm071
+ rorx rdx, rax, 3272
+%if %173
+ lea eax, [rax+rdx+1]74
+ shr eax, 175
+%else76
+ add eax, edx77
+%endif78
+%else79
+ HADDD m0, m180
+ movd eax, xm081
+%if %182
+ inc eax83
+ shr eax, 184
+%endif85
+%endif86
+ RET87
+%endmacro88
+89
+%macro HMAXABSW2 4 ; a, b, tmp1, tmp290
+ pabsw m%1, m%191
+ pabsw m%2, m%292
+ psrldq m%3, m%1, 293
+ psrld m%4, m%2, 1694
+ pmaxsw m%1, m%395
+ pmaxsw m%2, m%496
+%endmacro97
+%if HIGH_BIT_DEPTH==098
+INIT_ZMM avx51299
+cglobal pixel_satd_16x8_internal100
+ vbroadcasti64x4 m6, [hmul_16p]101
+ kxnorb k2, k2, k2102
+ mov r4d, 0x55555555103
+ knotw k2, k2104
+ kmovd k1, r4d105
+ lea r4, [3*r1]106
+ lea r5, [3*r3]107
+satd_16x8_avx512:108
+ vbroadcasti128 ym0, [r0]109
+ vbroadcasti32x4 m0 {k2}, [r0+4*r1] ; 0 0 4 4110
+ vbroadcasti128 ym4, [r2]111
+ vbroadcasti32x4 m4 {k2}, [r2+4*r3]112
+ vbroadcasti128 ym2, [r0+2*r1]113
+ vbroadcasti32x4 m2 {k2}, [r0+2*r4] ; 2 2 6 6114
+ vbroadcasti128 ym5, [r2+2*r3]115
+ vbroadcasti32x4 m5 {k2}, [r2+2*r5]116
+ DIFF_SUMSUB_SSSE3 0, 4, 2, 5, 6117
+ vbroadcasti128 ym1, [r0+r1]118
+ vbroadcasti128 ym4, [r2+r3]119
+ vbroadcasti128 ym3, [r0+r4]120
+ vbroadcasti128 ym5, [r2+r5]121
+ lea r0, [r0+4*r1]122
+ lea r2, [r2+4*r3]123
+ vbroadcasti32x4 m1 {k2}, [r0+r1] ; 1 1 5 5124
+ vbroadcasti32x4 m4 {k2}, [r2+r3]125
+ vbroadcasti32x4 m3 {k2}, [r0+r4] ; 3 3 7 7126
+ vbroadcasti32x4 m5 {k2}, [r2+r5]127
+ DIFF_SUMSUB_SSSE3 1, 4, 3, 5, 6128
+ HADAMARD4_V 0, 1, 2, 3, 4129
+ HMAXABSW2 0, 2, 4, 5130
+ HMAXABSW2 1, 3, 4, 5131
+ paddw m4, m0, m2 ; m1132
+ paddw m2, m1, m3 ; m0133
+ ret134
+135
+cglobal pixel_satd_8x8_internal136
+ vbroadcasti64x4 m4, [hmul_16p]137
+ mov r4d, 0x55555555138
+ kmovd k1, r4d ; 01010101139
+ kshiftlb k2, k1, 5 ; 10100000140
+ kshiftlb k3, k1, 4 ; 01010000141
+ lea r4, [3*r1]142
+ lea r5, [3*r3]143
+satd_8x8_avx512:144
+ SATD_AVX512_LOAD8 q, ym, k1, k2, k3 ; 2 0 2 0 6 4 6 4145
+ SATD_AVX512_PACKED ; 3 1 3 1 7 5 7 5146
+ ret147
+148
+cglobal pixel_satd_16x8, 4,6149
+ call pixel_satd_16x8_internal_avx512150
+ jmp satd_zmm_avx512_end151
+152
+cglobal pixel_satd_16x16, 4,6153
+ call pixel_satd_16x8_internal_avx512154
+ lea r0, [r0+4*r1]155
+ lea r2, [r2+4*r3]156
+ paddw m7, m0, m1157
+ call satd_16x8_avx512158
+ paddw m1, m7159
+ jmp satd_zmm_avx512_end160
+161
+cglobal pixel_satd_8x8, 4,6162
+ call pixel_satd_8x8_internal_avx512163
+satd_zmm_avx512_end:164
+ SATD_AVX512_END165
+166
+cglobal pixel_satd_8x16, 4,6167
+ call pixel_satd_8x8_internal_avx512168
+ lea r0, [r0+4*r1]169
+ lea r2, [r2+4*r3]170
+ paddw m5, m0, m1171
+ call satd_8x8_avx512172
+ paddw m1, m5173
+ jmp satd_zmm_avx512_end174
+175
+INIT_YMM avx512176
+cglobal pixel_satd_4x8_internal177
+ vbroadcasti128 m4, [hmul_4p]178
+ mov r4d, 0x55550c179
+ kmovd k2, r4d ; 00001100180
+ kshiftlb k3, k2, 2 ; 00110000181
+ kshiftlb k4, k2, 4 ; 11000000182
+ kshiftrd k1, k2, 8 ; 01010101183
+ lea r4, [3*r1]184
+ lea r5, [3*r3]185
+satd_4x8_avx512:186
+ SATD_AVX512_LOAD8 d, xm, k2, k3, k4 ; 0 0 2 2 4 4 6 6187
+satd_ymm_avx512: ; 1 1 3 3 5 5 7 7188
+ SATD_AVX512_PACKED189
+ ret190
+191
+cglobal pixel_satd_8x4, 4,5192
+ mova m4, [hmul_16p]193
+ mov r4d, 0x5555194
+ kmovw k1, r4d195
+ SATD_AVX512_LOAD4 q, k1 ; 2 0 2 0196
+ call satd_ymm_avx512 ; 3 1 3 1197
+ jmp satd_ymm_avx512_end2198
+199
+cglobal pixel_satd_4x8, 4,6200
+ call pixel_satd_4x8_internal_avx512201
+satd_ymm_avx512_end:202
+%if ARCH_X86_64 == 0203
+ pop r5d204
+ %assign regs_used 5205
+%endif206
+satd_ymm_avx512_end2:207
+ SATD_AVX512_END208
+209
+cglobal pixel_satd_4x16, 4,6210
+ call pixel_satd_4x8_internal_avx512211
+ lea r0, [r0+4*r1]212
+ lea r2, [r2+4*r3]213
+ paddw m5, m0, m1214
+ call satd_4x8_avx512215
+ paddw m1, m5216
+ jmp satd_ymm_avx512_end217
+218
+INIT_XMM avx512219
+cglobal pixel_satd_4x4, 4,5220
+ mova m4, [hmul_4p]221
+ mov r4d, 0x550c222
+ kmovw k2, r4d223
+ kshiftrw k1, k2, 8224
+ SATD_AVX512_LOAD4 d, k2 ; 0 0 2 2225
+ SATD_AVX512_PACKED ; 1 1 3 3226
+ SWAP 0, 1227
+ SATD_AVX512_END228
+229
+INIT_ZMM avx512230
+cglobal pixel_sa8d_8x8, 4,6231
+ vbroadcasti64x4 m4, [hmul_16p]232
+ mov r4d, 0x55555555233
+ kmovd k1, r4d ; 01010101234
+ kshiftlb k2, k1, 5 ; 10100000235
+ kshiftlb k3, k1, 4 ; 01010000236
+ lea r4, [3*r1]237
+ lea r5, [3*r3]238
+ SATD_AVX512_LOAD8 q, ym, k1, k2, k3 ; 2 0 2 0 6 4 6 4239
+ DIFF_SUMSUB_SSSE3 0, 2, 1, 3, 4 ; 3 1 3 1 7 5 7 5240
+ SUMSUB_BA w, 0, 1, 2241
+ SBUTTERFLY qdq, 0, 1, 2242
+ SUMSUB_BA w, 0, 1, 2243
+ shufps m2, m0, m1, q2020244
+ shufps m1, m0, m1, q3131245
+ SUMSUB_BA w, 2, 1, 0246
+ vshufi32x4 m0, m2, m1, q1010247
+ vshufi32x4 m1, m2, m1, q3232248
+ SUMSUB_BA w, 0, 1, 2249
+ HMAXABSW2 0, 1, 2, 3250
+ SATD_AVX512_END 1251
+%endif252
; Input 10bit, Output 8bit253
;------------------------------------------------------------------------------------------------------------------------254
;void planecopy_sc(uint16_t *src, intptr_t srcStride, pixel *dst, intptr_t dstStride, int width, int height, int shift, uint16_t mask)255
256
257
.end:258
RET259
+INIT_ZMM avx512260
+cglobal upShift_16, 4,7,4261
+ mov r4d, r4m262
+ mov r5d, r5m263
+ movd xm0, r6m ; m0 = shift264
+ vbroadcasti32x4 m3, [pw_pixel_max]265
+ FIX_STRIDES r1d, r3d266
+ dec r5d267
+.loopH:268
+ xor r6d, r6d269
+.loopW:270
+ movu m1, [r0 + r6 * SIZEOF_PIXEL]271
+ psllw m1, xm0272
+ pand m1, m3273
+ movu [r2 + r6 * SIZEOF_PIXEL], m1274
+275
+ add r6, mmsize / SIZEOF_PIXEL276
+ cmp r6d, r4d277
+ jl .loopW278
+279
+ ; move to next row280
+ add r0, r1281
+ add r2, r3282
+ dec r5d283
+ jnz .loopH284
285
+ ; processing last row of every frame [To handle width which not a multiple of 32]286
287
+.loop32:288
+ movu m1, [r0 + (r4 - mmsize/2) * 2]289
+ psllw m1, xm0290
+ pand m1, m3291
+ movu [r2 + (r4 - mmsize/2) * 2], m1292
+293
+ sub r4d, mmsize/2294
+ jz .end295
+ cmp r4d, mmsize/2296
+ jge .loop32297
+298
+ ; process partial pixels299
+ movu m1, [r0]300
+ psllw m1, xm0301
+ pand m1, m3302
+ movu [r2], m1303
+304
+.end:305
+ RET306
;---------------------------------------------------------------------------------------------------------------------307
;int psyCost_pp(const pixel* source, intptr_t sstride, const pixel* recon, intptr_t rstride)308
;---------------------------------------------------------------------------------------------------------------------309
310
pabsd m11, m11311
%endmacro312
313
+%macro PSY_COST_PP_8x8_AVX512_MAIN12 0314
+ ; load source and recon pixels315
+ lea r4, [r1 * 3]316
+ pmovzxwd ym0, [r0]317
+ pmovzxwd ym1, [r0 + r1]318
+ pmovzxwd ym2, [r0 + r1 * 2]319
+ pmovzxwd ym3, [r0 + r4]320
+ lea r5, [r0 + r1 * 4]321
+ pmovzxwd ym4, [r5]322
+ pmovzxwd ym5, [r5 + r1]323
+ pmovzxwd ym6, [r5 + r1 * 2]324
+ pmovzxwd ym7, [r5 + r4]325
+326
+ lea r4, [r3 * 3]327
+ pmovzxwd ym16, [r2]328
+ pmovzxwd ym17, [r2 + r3]329
+ pmovzxwd ym18, [r2 + r3 * 2]330
+ pmovzxwd ym19, [r2 + r4]331
+ lea r5, [r2 + r3 * 4]332
+ pmovzxwd ym20, [r5]333
+ pmovzxwd ym21, [r5 + r3]334
+ pmovzxwd ym22, [r5 + r3 * 2]335
+ pmovzxwd ym23, [r5 + r4]336
+337
+ vinserti64x4 m0, m0, ym16, 1338
+ vinserti64x4 m1, m1, ym17, 1339
+ vinserti64x4 m2, m2, ym18, 1340
+ vinserti64x4 m3, m3, ym19, 1341
+ vinserti64x4 m4, m4, ym20, 1342
+ vinserti64x4 m5, m5, ym21, 1343
+ vinserti64x4 m6, m6, ym22, 1344
+ vinserti64x4 m7, m7, ym23, 1345
+346
+ ; source + recon SAD347
+ paddd m8, m0, m1348
+ paddd m8, m2349
+ paddd m8, m3350
+ paddd m8, m4351
+ paddd m8, m5352
+ paddd m8, m6353
+ paddd m8, m7354
+355
+ vextracti64x4 ym15, m8, 1356
+357
+ vextracti128 xm9, ym8, 1358
+ paddd ym8, ym9 ; sad_8x8359
+ movhlps xm9, xm8360
+ paddd xm8, xm9361
+ pshuflw xm9, xm8, 0Eh362
+ paddd xm8, xm9363
+ psrld ym8, 2364
+365
+ vextracti128 xm9, ym15, 1366
+ paddd ym15, ym9 ; sad_8x8367
+ movhlps xm9, xm15368
+ paddd xm15, xm9369
+ pshuflw xm9, xm15, 0Eh370
+ paddd xm15, xm9371
+ psrld ym15, 2372
+373
+ ; source and recon SA8D374
+ psubd m9, m1, m0375
+ paddd m0, m1376
+ psubd m1, m3, m2377
+ paddd m2, m3378
+ punpckhdq m3, m0, m9379
+ punpckldq m0, m9380
+ psubd m9, m3, m0381
+ paddd m0, m3382
+ punpckhdq m3, m2, m1383
+ punpckldq m2, m1384
+ psubd m10, m3, m2385
+ paddd m2, m3386
+ psubd m3, m5, m4387
+ paddd m4, m5388
+ psubd m5, m7, m6389
+ paddd m6, m7390
+ punpckhdq m1, m4, m3391
+ punpckldq m4, m3392
+ psubd m7, m1, m4393
+ paddd m4, m1394
+ punpckhdq m3, m6, m5395
+ punpckldq m6, m5396
+ psubd m1, m3, m6397
+ paddd m6, m3398
+ psubd m3, m2, m0399
+ paddd m0, m2400
+ psubd m2, m10, m9401
+ paddd m9, m10402
+ punpckhqdq m5, m0, m3403
+ punpcklqdq m0, m3404
+ psubd m10, m5, m0405
+ paddd m0, m5406
+ punpckhqdq m3, m9, m2407
+ punpcklqdq m9, m2408
+ psubd m5, m3, m9409
+ paddd m9, m3410
+ psubd m3, m6, m4411
+ paddd m4, m6412
+ psubd m6, m1, m7413
+ paddd m7, m1414
+ punpckhqdq m2, m4, m3415
+ punpcklqdq m4, m3416
+ psubd m1, m2, m4417
+ paddd m4, m2418
+ punpckhqdq m3, m7, m6419
+ punpcklqdq m7, m6420
+421
+ psubd m2, m3, m7422
+ paddd m7, m3423
+ psubd m3, m4, m0424
+ paddd m0, m4425
+ psubd m4, m1, m10426
+ paddd m10, m1427
+428
+ mova m16, m13429
+ mova m17, m14430
+ vpermi2q m16, m0, m3431
+ vpermi2q m17, m0, m3432
+433
+ pabsd m17, m17434
+ pabsd m16, m16435
+ pmaxsd m17, m16436
+437
+ mova m18, m13438
+ mova m19, m14439
+ vpermi2q m18, m10, m4440
+ vpermi2q m19, m10, m4441
+442
+ pabsd m19, m19443
+ pabsd m18, m18444
+ pmaxsd m19, m18445
+ psubd m18, m7, m9446
+ paddd m9, m7447
+ psubd m7, m2, m5448
+ paddd m5, m2449
+450
+ mova m20, m13451
+ mova m21, m14452
+ vpermi2q m20, m9, m18453
+ vpermi2q m21, m9, m18454
+455
+ pabsd m21, m21456
+ pabsd m20, m20457
+ pmaxsd m21, m20458
+459
+ mova m22, m13460
+ mova m23, m14461
+ vpermi2q m22, m5, m7462
+ vpermi2q m23, m5, m7463
+464
+ pabsd m23, m23465
+ pabsd m22, m22466
+ pmaxsd m23, m22467
+ paddd m17, m21468
+ paddd m17, m19469
+ paddd m17, m23470
+471
+ vextracti64x4 ym26, m17, 1472
+473
+ vextracti128 xm9, m17, 1474
+ paddd ym17, ym9 ; sad_8x8475
+ movhlps xm9, xm17476
+ paddd xm17, xm9477
+ pshuflw xm9, xm17, 0Eh478
+ paddd xm17, xm9479
+ paddd ym17, [pd_1]480
+ psrld ym17, 1 ; sa8d_8x8481
+482
+ vextracti128 xm9, ym26, 1483
+ paddd ym26, ym9 ; sad_8x8484
+ movhlps xm9, xm26485
+ paddd xm26, xm9486
+ pshuflw xm9, xm26, 0Eh487
+ paddd xm26, xm9488
+ paddd ym26, [pd_1]489
+ psrld ym26, 1 ; sa8d_8x8490
+491
+492
+493
+ psubd ym11, ym17, ym8 ; sa8d_8x8 - sad_8x8494
+ psubd ym12, ym26, ym15 ; sa8d_8x8 - sad_8x8495
+496
+ psubd ym11, ym12497
+ pabsd ym11, ym11498
+%endmacro499
+500
+%macro PSY_PP_INPUT_AVX512_MAIN10 0501
+ lea r4, [r1 * 3]502
+ movu xm0, [r0]503
+ movu xm1, [r0 + r1]504
+ movu xm2, [r0 + r1 * 2]505
+ movu xm3, [r0 + r4]506
+ lea r5, [r0 + r1 * 4]507
+ movu xm4, [r5]508
+ movu xm5, [r5 + r1]509
+ movu xm6, [r5 + r1 * 2]510
+ movu xm7, [r5 + r4]511
+512
+ lea r4, [r3 * 3]513
+ vinserti128 ym0, ym0, [r2], 1514
+ vinserti128 ym1, ym1, [r2 + r3], 1515
+ vinserti128 ym2, ym2, [r2 + r3 * 2], 1516
+ vinserti128 ym3, ym3, [r2 + r4], 1517
+ lea r5, [r2 + r3 * 4]518
+ vinserti128 ym4, ym4, [r5], 1519
+ vinserti128 ym5, ym5, [r5 + r3], 1520
+ vinserti128 ym6, ym6, [r5 + r3 * 2], 1521
+ vinserti128 ym7, ym7, [r5 + r4], 1522
+523
+ add r0, 16524
+ add r2, 16525
+526
+ lea r4, [r1 * 3]527
+ vinserti32x4 m0, m0, [r0], 2528
+ vinserti32x4 m1, m1, [r0 + r1], 2529
+ vinserti32x4 m2, m2, [r0 + r1 * 2], 2530
+ vinserti32x4 m3, m3, [r0 + r4], 2531
+ lea r5, [r0 + r1 * 4]532
+ vinserti32x4 m4, m4, [r5], 2533
+ vinserti32x4 m5, m5, [r5 + r1], 2534
+ vinserti32x4 m6, m6, [r5 + r1 * 2], 2535
+ vinserti32x4 m7, m7, [r5 + r4], 2536
+537
+ lea r4, [r3 * 3]538
+ vinserti32x4 m0, m0, [r2], 3539
+ vinserti32x4 m1, m1, [r2 + r3], 3540
+ vinserti32x4 m2, m2, [r2 + r3 * 2], 3541
+ vinserti32x4 m3, m3, [r2 + r4], 3542
+ lea r5, [r2 + r3 * 4]543
+ vinserti32x4 m4, m4, [r5], 3544
+ vinserti32x4 m5, m5, [r5 + r3], 3545
+ vinserti32x4 m6, m6, [r5 + r3 * 2], 3546
+ vinserti32x4 m7, m7, [r5 + r4], 3547
+%endmacro548
+549
+550
+%macro PSY_PP_16x8_AVX512_MAIN10 0551
+ paddw m8, m0, m1552
+ paddw m8, m2553
+ paddw m8, m3554
+ paddw m8, m4555
+ paddw m8, m5556
+ paddw m8, m6557
+ paddw m8, m7558
+ pmaddwd m8, m14559
+560
+ psrldq m9, m8, 8561
+ paddd m8, m9562
+ psrldq m9, m8, 4563
+ paddd m8, m9564
+ psrld m8, 2565
+566
+ psubw m9, m1, m0567
+ paddw m0, m1568
+ psubw m1, m3, m2569
+ paddw m2, m3570
+ punpckhwd m3, m0, m9571
+ punpcklwd m0, m9572
+ psubw m9, m3, m0573
+ paddw m0, m3574
+ punpckhwd m3, m2, m1575
+ punpcklwd m2, m1576
+ psubw m10, m3, m2577
+ paddw m2, m3578
+579
+ psubw m3, m5, m4580
+ paddw m4, m5581
+ psubw m5, m7, m6582
+ paddw m6, m7583
+ punpckhwd m1, m4, m3584
+ punpcklwd m4, m3585
+ psubw m7, m1, m4586
+ paddw m4, m1587
+ punpckhwd m3, m6, m5588
+ punpcklwd m6, m5589
+ psubw m1, m3, m6590
+ paddw m6, m3591
+592
+ psubw m3, m2, m0593
+ paddw m0, m2594
+ psubw m2, m10, m9595
+ paddw m9, m10596
+ punpckhdq m5, m0, m3597
+ punpckldq m0, m3598
+ psubw m10, m5, m0599
+ paddw m0, m5600
+ punpckhdq m3, m9, m2601
+ punpckldq m9, m2602
+ psubw m5, m3, m9603
+ paddw m9, m3604
+605
+ psubw m3, m6, m4606
+ paddw m4, m6607
+ psubw m6, m1, m7608
+ paddw m7, m1609
+ punpckhdq m2, m4, m3610
+ punpckldq m4, m3611
+ psubw m1, m2, m4612
+ paddw m4, m2613
+ punpckhdq m3, m7, m6614
+ punpckldq m7, m6615
+ psubw m2, m3, m7616
+ paddw m7, m3617
+618
+ psubw m3, m4, m0619
+ paddw m0, m4620
+ psubw m4, m1, m10621
+ paddw m10, m1622
+ punpckhqdq m6, m0, m3623
+ punpcklqdq m0, m3624
+ pabsw m0, m0625
+ pabsw m6, m6626
+ pmaxsw m0, m6627
+ punpckhqdq m3, m10, m4628
+ punpcklqdq m10, m4629
+ pabsw m10, m10630
+ pabsw m3, m3631
+ pmaxsw m10, m3632
+633
+ psubw m3, m7, m9634
+ paddw m9, m7635
+ psubw m7, m2, m5636
+ paddw m5, m2637
+ punpckhqdq m4, m9, m3638
+ punpcklqdq m9, m3639
+ pabsw m9, m9640
+ pabsw m4, m4641
+ pmaxsw m9, m4642
+ punpckhqdq m3, m5, m7643
+ punpcklqdq m5, m7644
+ pabsw m5, m5645
+ pabsw m3, m3646
+ pmaxsw m5, m3647
+648
+ paddd m0, m9649
+ paddd m0, m10650
+ paddd m0, m5651
+ psrld m9, m0, 16652
+ pslld m0, 16653
+ psrld m0, 16654
+ paddd m0, m9655
+ psrldq m9, m0, 8656
+ paddd m0, m9657
+ psrldq m9, m0, 4658
+ paddd m0, m9659
+ paddd m0, m15660
+ psrld m0, 1661
+ psubd m0, m8662
+663
+ vextracti64x4 ym2, m0, 1664
+665
+ vextracti128 xm3, ym2, 1666
+ psubd xm3, xm2667
+ pabsd xm3, xm3668
+669
+ vextracti128 xm1, ym0, 1670
+ psubd xm1, xm0671
+ pabsd xm1, xm1672
+ paddd xm1, xm3673
+%endmacro674
+675
+%macro PSY_PP_INPUT_AVX512_MAIN 0676
+ movu xm16, [r0 + r1 * 0]677
+ movu xm17, [r0 + r1 * 1]678
+ movu xm18, [r0 + r1 * 2]679
+ movu xm19, [r0 + r4 * 1]680
+681
+ movu xm20, [r2 + r3 * 0]682
+ movu xm21, [r2 + r3 * 1]683
+ movu xm22, [r2 + r3 * 2]684
+ movu xm23, [r2 + r7 * 1]685
+686
+ mova m0, m26687
+ vpermi2q m0, m16, m20688
+ mova m1, m26689
+ vpermi2q m1, m17, m21690
+ mova m2, m26691
+ vpermi2q m2, m18, m22692
+ mova m3, m26693
+ vpermi2q m3, m19, m23694
+695
+696
+ lea r5, [r0 + r1 * 4]697
+ lea r6, [r2 + r3 * 4]698
+699
+ movu xm16, [r5 + r1 * 0]700
+ movu xm17, [r5 + r1 * 1]701
+ movu xm18, [r5 + r1 * 2]702
+ movu xm19, [r5 + r4 * 1]703
+704
+ movu xm20, [r6 + r3 * 0]705
+ movu xm21, [r6 + r3 * 1]706
+ movu xm22, [r6 + r3 * 2]707
+ movu xm23, [r6 + r7 * 1]708
+709
+ mova m4, m26710
+ vpermi2q m4, m16, m20711
+ mova m5, m26712
+ vpermi2q m5, m17, m21713
+ mova m6, m26714
+ vpermi2q m6, m18, m22715
+ mova m7, m26716
+ vpermi2q m7, m19, m23717
+%endmacro718
+719
+%macro PSY_PP_16x8_AVX512_MAIN 0720
+ pmaddubsw m0, m8721
+ pmaddubsw m1, m8722
+ pmaddubsw m2, m8723
+ pmaddubsw m3, m8724
+ pmaddubsw m4, m8725
+ pmaddubsw m5, m8726
+ pmaddubsw m6, m8727
+ pmaddubsw m7, m8728
+729
+ paddw m11, m0, m1730
+ paddw m11, m2731
+ paddw m11, m3732
+ paddw m11, m4733
+ paddw m11, m5734
+ paddw m11, m6735
+ paddw m11, m7736
+737
+ pmaddwd m11, m14738
+ psrldq m10, m11, 4739
+ paddd m11, m10740
+ psrld m11, 2741
+742
+ mova m9, m0743
+ paddw m0, m1744
+ psubw m1, m9745
+ mova m9, m2746
+ paddw m2, m3747
+ psubw m3, m9748
+ mova m9, m0749
+ paddw m0, m2750
+ psubw m2, m9751
+ mova m9, m1752
+ paddw m1, m3753
+ psubw m3, m9754
+755
+ movdqa m9, m4756
+ paddw m4, m5757
+ psubw m5, m9758
+ movdqa m9, m6759
+ paddw m6, m7760
+ psubw m7, m9761
+ movdqa m9, m4762
+ paddw m4, m6763
+ psubw m6, m9764
+ movdqa m9, m5765
+ paddw m5, m7766
+ psubw m7, m9767
+768
+ movdqa m9, m0769
+ paddw m0, m4770
+ psubw m4, m9771
+ movdqa m9, m1772
+ paddw m1, m5773
+ psubw m5, m9774
+775
+ mova m9, m0776
+ vshufps m9, m9, m4, 11011101b777
+ vshufps m0, m0, m4, 10001000b778
+779
+ movdqa m4, m0780
+ paddw m16, m0, m9781
+ psubw m17, m9, m4782
+783
+ movaps m4, m1784
+ vshufps m4, m4, m5, 11011101b785
+ vshufps m1, m1, m5, 10001000b786
+787
+ movdqa m5, m1788
+ paddw m18, m1, m4789
+ psubw m19, m4, m5790
+791
+ movdqa m5, m2792
+ paddw m2, m6793
+ psubw m6, m5794
+ movdqa m5, m3795
+ paddw m3, m7796
+ psubw m7, m5797
+798
+ movaps m5, m2799
+ vshufps m5, m5, m6, 11011101b800
+ vshufps m2, m2, m6, 10001000b801
+802
+ movdqa m6, m2803
+ paddw m20, m2, m5804
+ psubw m21, m5, m6805
+806
+ movaps m6, m3807
+808
+ vshufps m6, m6, m7, 11011101b809
+ vshufps m3, m3, m7, 10001000b810
+811
+ movdqa m7, m3812
+ paddw m22, m3, m6813
+ psubw m23, m6, m7814
+815
+ movdqa m7, m16816
+817
+ vextracti64x4 ym24, m16, 1818
+ vextracti64x4 ym25, m17, 1819
+ pblendw ym16, ym17, 10101010b820
+ pblendw ym24, ym25, 10101010b821
+ vinserti64x4 m16, m16, ym24, 1822
+823
+ pslld m17, 10h824
+ psrld m7, 10h825
+ por m17, m7826
+ pabsw m16, m16827
+ pabsw m17, m17828
+ pmaxsw m16, m17829
+ movdqa m7, m18830
+831
+ vextracti64x4 ym24, m18, 1832
+ vextracti64x4 ym25, m19, 1833
+ pblendw ym18, ym19, 10101010b834
+ pblendw ym24, ym25, 10101010b835
+ vinserti64x4 m18, m18, ym24, 1836
+837
+ pslld m19, 10h838
+ psrld m7, 10h839
+ por m19, m7840
+ pabsw m18, m18841
+ pabsw m19, m19842
+ pmaxsw m18, m19843
+ movdqa m7, m20844
+845
+ vextracti64x4 ym24, m20, 1846
+ vextracti64x4 ym25, m21, 1847
+ pblendw ym20, ym21, 10101010b848
+ pblendw ym24, ym25, 10101010b849
+ vinserti64x4 m20, m20, ym24, 1850
+851
+ pslld m21, 10h852
+ psrld m7, 10h853
+ por m21, m7854
+ pabsw m20, m20855
+ pabsw m21, m21856
+ pmaxsw m20, m21857
+ mova m7, m22858
+859
+ vextracti64x4 ym24, m22, 1860
+ vextracti64x4 ym25, m23, 1861
+ pblendw ym22, ym23, 10101010b862
+ pblendw ym24, ym25, 10101010b863
+ vinserti64x4 m22, m22, ym24, 1864
+865
+ pslld m23, 10h866
+ psrld m7, 10h867
+ por m23, m7868
+ pabsw m22, m22869
+ pabsw m23, m23870
+ pmaxsw m22, m23871
+ paddw m16, m18872
+ paddw m16, m20873
+ paddw m16, m22874
+ pmaddwd m16, m14875
+ psrldq m1, m16, 8876
+ paddd m16, m1877
+878
+ pshuflw m1, m16, 00001110b879
+ paddd m16, m1880
+ paddd m16, m15881
+ psrld m16, 1882
+883
+ psubd m16, m11884
+ vextracti64x4 ym2, m16, 1885
+886
+ vextracti128 xm1, ym16, 1887
+ psubd xm16, xm1888
+ pabsd xm16, xm16889
+890
+ vextracti128 xm3, ym2, 1891
+ psubd xm3, xm2892
+ pabsd xm3, xm3893
+ paddd xm16, xm3894
+%endmacro895
+896
+897
%if ARCH_X86_64898
INIT_YMM avx2899
%if HIGH_BIT_DEPTH && BIT_DEPTH == 12900
901
RET902
%endif903
%endif904
+%if ARCH_X86_64905
+INIT_ZMM avx512906
+%if HIGH_BIT_DEPTH && BIT_DEPTH == 12907
+cglobal psyCost_pp_16x16, 4, 10, 27908
+ add r1d, r1d909
+ add r3d, r3d910
+ pxor m24, m24911
+ movu m13, [psy_pp_shuff1]912
+ movu m14, [psy_pp_shuff2]913
+914
+ mov r8d, 2915
+.loopH:916
+ mov r9d, 2917
+.loopW:918
+ PSY_COST_PP_8x8_AVX512_MAIN12919
+920
+ paddd xm24, xm11921
+ add r0, 16922
+ add r2, 16923
+ dec r9d924
+ jnz .loopW925
+ lea r0, [r0 + r1 * 8 - 32]926
+ lea r2, [r2 + r3 * 8 - 32]927
+ dec r8d928
+ jnz .loopH929
+ movd eax, xm24930
+ RET931
+%endif932
+933
+%if HIGH_BIT_DEPTH && BIT_DEPTH == 10934
+cglobal psyCost_pp_16x16, 4, 10, 16935
+ add r1d, r1d936
+ add r3d, r3d937
+ pxor m11, m11938
+ vbroadcasti32x8 m14, [pw_1]939
+ vbroadcasti32x8 m15, [pd_1]940
+941
+ mov r8d, 2942
+.loopH:943
+ PSY_PP_INPUT_AVX512_MAIN10944
+ PSY_PP_16x8_AVX512_MAIN10945
+946
+ paddd xm11, xm1947
+ lea r0, [r0 + r1 * 8 - 16]948
+ lea r2, [r2 + r3 * 8 - 16]949
+ dec r8d950
+ jnz .loopH951
+ movd eax, xm11952
+ RET953
+%endif954
+955
+%if BIT_DEPTH == 8956
+cglobal psyCost_pp_16x16, 4, 10, 27957
+ lea r4, [3 * r1]958
+ lea r7, [3 * r3]959
+ vbroadcasti32x8 m8, [hmul_8p]960
+ pxor m13, m13961
+ vbroadcasti32x8 m14, [pw_1]962
+ vbroadcasti32x8 m15, [pd_1]963
+ movu m26, [psy_pp_shuff3]964
+965
+ mov r8d, 2966
+.loopH:967
+ PSY_PP_INPUT_AVX512_MAIN968
+ PSY_PP_16x8_AVX512_MAIN969
+970
+ paddd m13, m16971
+ lea r0, [r0 + r1 * 8]972
+ lea r2, [r2 + r3 * 8]973
+ dec r8d974
+ jnz .loopH975
+ movd eax, xm13976
+ RET977
+%endif978
+%endif979
+980
+%if ARCH_X86_64981
+INIT_ZMM avx512982
+%if HIGH_BIT_DEPTH && BIT_DEPTH == 12983
+cglobal psyCost_pp_32x32, 4, 10, 27984
+ add r1d, r1d985
+ add r3d, r3d986
+ pxor m24, m24987
+ movu m13, [psy_pp_shuff1]988
+ movu m14, [psy_pp_shuff2]989
+990
+ mov r8d, 4991
+.loopH:992
+ mov r9d, 4993
+.loopW:994
+ PSY_COST_PP_8x8_AVX512_MAIN12995
+996
+ paddd xm24, xm11997
+ add r0, 16998
+ add r2, 16999
+ dec r9d1000
+ jnz .loopW1001
+ lea r0, [r0 + r1 * 8 - 64]1002
+ lea r2, [r2 + r3 * 8 - 64]1003
+ dec r8d1004
+ jnz .loopH1005
+ movd eax, xm241006
+ RET1007
+%endif1008
+1009
+%if HIGH_BIT_DEPTH && BIT_DEPTH == 101010
+cglobal psyCost_pp_32x32, 4, 10, 161011
+ add r1d, r1d1012
+ add r3d, r3d1013
+ pxor m11, m111014
+ vbroadcasti32x8 m14, [pw_1]1015
+ vbroadcasti32x8 m15, [pd_1]1016
+1017
+ mov r8d, 41018
+.loopH:1019
+ mov r9d, 21020
+.loopW:1021
+ PSY_PP_INPUT_AVX512_MAIN101022
+ PSY_PP_16x8_AVX512_MAIN101023
+1024
+ paddd xm11, xm11025
+ add r0, 161026
+ add r2, 161027
+ dec r9d1028
+ jnz .loopW1029
+ lea r0, [r0 + r1 * 8 - 64]1030
+ lea r2, [r2 + r3 * 8 - 64]1031
+ dec r8d1032
+ jnz .loopH1033
+ movd eax, xm111034
+ RET1035
+%endif1036
+1037
+%if BIT_DEPTH == 81038
+cglobal psyCost_pp_32x32, 4, 10, 271039
+ lea r4, [3 * r1]1040
+ lea r7, [3 * r3]1041
+ vbroadcasti32x8 m8, [hmul_8p]1042
+ pxor m13, m131043
+ vbroadcasti32x8 m14, [pw_1]1044
+ vbroadcasti32x8 m15, [pd_1]1045
+ movu m26, [psy_pp_shuff3]1046
+1047
+ mov r8d, 41048
+.loopH:1049
+ mov r9d, 21050
+.loopW:1051
+ PSY_PP_INPUT_AVX512_MAIN1052
+ PSY_PP_16x8_AVX512_MAIN1053
+1054
+ paddd m13, m161055
+ add r0, 161056
+ add r2, 161057
+ dec r9d1058
+ jnz .loopW1059
+ lea r0, [r0 + r1 * 8 - 32]1060
+ lea r2, [r2 + r3 * 8 - 32]1061
+ dec r8d1062
+ jnz .loopH1063
+ movd eax, xm131064
+ RET1065
+%endif1066
+%endif1067
+1068
+%if ARCH_X86_641069
+INIT_ZMM avx5121070
+%if HIGH_BIT_DEPTH && BIT_DEPTH == 121071
+cglobal psyCost_pp_64x64, 4, 10, 271072
+ add r1d, r1d1073
+ add r3d, r3d1074
+ pxor m24, m241075
+ movu m13, [psy_pp_shuff1]1076
+ movu m14, [psy_pp_shuff2]1077
+1078
+ mov r8d, 81079
+.loopH:1080
+ mov r9d, 81081
+.loopW:1082
+ PSY_COST_PP_8x8_AVX512_MAIN121083
+1084
+ paddd xm24, xm111085
+ add r0, 161086
+ add r2, 161087
+ dec r9d1088
+ jnz .loopW1089
+ lea r0, [r0 + r1 * 8 - 128]1090
+ lea r2, [r2 + r3 * 8 - 128]1091
+ dec r8d1092
+ jnz .loopH1093
+ movd eax, xm241094
+ RET1095
+%endif1096
+1097
+%if HIGH_BIT_DEPTH && BIT_DEPTH == 101098
+cglobal psyCost_pp_64x64, 4, 10, 161099
+ add r1d, r1d1100
+ add r3d, r3d1101
+ pxor m11, m111102
+ vbroadcasti32x8 m14, [pw_1]1103
+ vbroadcasti32x8 m15, [pd_1]1104
+1105
+ mov r8d, 81106
+.loopH:1107
+ mov r9d, 41108
+.loopW:1109
+ PSY_PP_INPUT_AVX512_MAIN101110
+ PSY_PP_16x8_AVX512_MAIN101111
+1112
+ paddd xm11, xm11113
+ add r0, 161114
+ add r2, 161115
+ dec r9d1116
+ jnz .loopW1117
+ lea r0, [r0 + r1 * 8 - 128]1118
+ lea r2, [r2 + r3 * 8 - 128]1119
+ dec r8d1120
+ jnz .loopH1121
+ movd eax, xm111122
+ RET1123
+%endif1124
+1125
+%if BIT_DEPTH == 81126
+cglobal psyCost_pp_64x64, 4, 10, 271127
+ lea r4, [3 * r1]1128
+ lea r7, [3 * r3]1129
+ vbroadcasti32x8 m8, [hmul_8p]1130
+ pxor m13, m131131
+ vbroadcasti32x8 m14, [pw_1]1132
+ vbroadcasti32x8 m15, [pd_1]1133
+ movu m26, [psy_pp_shuff3]1134
+1135
+ mov r8d, 81136
+.loopH:1137
+ mov r9d, 41138
+.loopW:1139
+ PSY_PP_INPUT_AVX512_MAIN1140
+ PSY_PP_16x8_AVX512_MAIN1141
+1142
+ paddd m13, m161143
+ add r0, 161144
+ add r2, 161145
+ dec r9d1146
+ jnz .loopW1147
+ lea r0, [r0 + r1 * 8 - 64]1148
+ lea r2, [r2 + r3 * 8 - 64]1149
+ dec r8d1150
+ jnz .loopH1151
+ movd eax, xm131152
+ RET1153
+%endif1154
+%endif1155
1156
;---------------------------------------------------------------------------------------------------------------------1157
;int psyCost_ss(const int16_t* source, intptr_t sstride, const int16_t* recon, intptr_t rstride)1158
1159
paddd xm0, xm11160
movd eax, xm01161
RET1162
-%endif ; ARCH_X86_64 == 1 && HIGH_BIT_DEPTH == 01163
1164
+%macro PROCESS_SATD_32x4_AVX512 0 ; function to compute satd cost for 32 columns, 4 rows1165
+ ; rows 0-31166
+ pmovzxbw m0, [r0]1167
+ pmovzxbw m4, [r2]1168
+ psubw m0, m41169
+ pmovzxbw m1, [r0 + r1]1170
+ pmovzxbw m5, [r2 + r3]1171
+ psubw m1, m51172
+ pmovzxbw m2, [r0 + r1 * 2]1173
+ pmovzxbw m4, [r2 + r3 * 2]1174
+ psubw m2, m41175
+ pmovzxbw m3, [r0 + r4]1176
+ pmovzxbw m5, [r2 + r5]1177
+ psubw m3, m51178
+ paddw m4, m0, m11179
+ psubw m1, m01180
+ paddw m0, m2, m31181
+ psubw m3, m21182
+ punpckhwd m2, m4, m11183
+ punpcklwd m4, m11184
+ punpckhwd m1, m0, m31185
+ punpcklwd m0, m31186
+ paddw m3, m4, m01187
+ psubw m0, m41188
+ paddw m4, m2, m11189
+ psubw m1, m21190
+ punpckhdq m2, m3, m01191
+ punpckldq m3, m01192
+ paddw m0, m3, m21193
+ psubw m2, m31194
+ punpckhdq m3, m4, m11195
+ punpckldq m4, m11196
+ paddw m1, m4, m31197
+ psubw m3, m41198
+ punpckhqdq m4, m0, m11199
+ punpcklqdq m0, m11200
+ pabsw m0, m01201
+ pabsw m4, m41202
+ pmaxsw m0, m0, m41203
+ punpckhqdq m1, m2, m31204
+ punpcklqdq m2, m31205
+ pabsw m2, m21206
+ pabsw m1, m11207
+ pmaxsw m2, m11208
+ pxor m7, m71209
+ mova m1, m01210
+ punpcklwd m1, m71211
+ paddd m6, m11212
+ mova m1, m01213
+ punpckhwd m1, m71214
+ paddd m6, m11215
+ pxor m7, m71216
+ mova m1, m21217
+ punpcklwd m1, m71218
+ paddd m6, m11219
+ mova m1, m21220
+ punpckhwd m1, m71221
+ paddd m6, m11222
+%endmacro1223
+1224
+%macro SATD_MAIN_AVX512_END 01225
+ vextracti32x8 ym7, m6, 11226
+ paddd ym6, ym71227
+ vextracti128 xm7, ym6, 11228
+ paddd xm6, xm6, xm71229
+ punpckhqdq xm7, xm6, xm61230
+ paddd xm6, xm71231
+ movq rax, xm61232
+ rorx rdx, rax, 321233
+ add eax, edx1234
+%endmacro1235
+1236
+%macro SATD_32xN_AVX512 11237
+INIT_ZMM avx5121238
+cglobal pixel_satd_32x%1, 4,6,81239
+ lea r4, [3 * r1]1240
+ lea r5, [3 * r3]1241
+ pxor m6, m61242
+%rep %1/4 - 11243
+ PROCESS_SATD_32x4_AVX5121244
+ lea r0, [r0 + 4 * r1]1245
+ lea r2, [r2 + 4 * r3]1246
+%endrep1247
+ PROCESS_SATD_32x4_AVX5121248
+ SATD_MAIN_AVX512_END1249
+ RET1250
+%endmacro1251
+1252
+SATD_32xN_AVX512 81253
+SATD_32xN_AVX512 161254
+SATD_32xN_AVX512 241255
+SATD_32xN_AVX512 321256
+SATD_32xN_AVX512 481257
+SATD_32xN_AVX512 641258
+1259
+%macro SATD_64xN_AVX512 11260
+INIT_ZMM avx5121261
+cglobal pixel_satd_64x%1, 4,8,81262
+ lea r4, [3 * r1]1263
+ lea r5, [3 * r3]1264
+ pxor m6, m61265
+ mov r6, r01266
+ mov r7, r21267
+1268
+%rep %1/4 - 11269
+ PROCESS_SATD_32x4_AVX5121270
+ lea r0, [r0 + 4 * r1]1271
+ lea r2, [r2 + 4 * r3]1272
+%endrep1273
+ PROCESS_SATD_32x4_AVX5121274
+ lea r0, [r6 + mmsize/2]1275
+ lea r2, [r7 + mmsize/2]1276
+%rep %1/4 - 11277
+ PROCESS_SATD_32x4_AVX5121278
+ lea r0, [r0 + 4 * r1]1279
+ lea r2, [r2 + 4 * r3]1280
+%endrep1281
+ PROCESS_SATD_32x4_AVX5121282
+ SATD_MAIN_AVX512_END1283
+ RET1284
+%endmacro1285
+1286
+SATD_64xN_AVX512 161287
+SATD_64xN_AVX512 321288
+SATD_64xN_AVX512 481289
+SATD_64xN_AVX512 641290
+%endif ; ARCH_X86_64 == 1 && HIGH_BIT_DEPTH == 01291
%if ARCH_X86_64 == 1 && HIGH_BIT_DEPTH == 11292
INIT_YMM avx21293
cglobal calc_satd_16x8 ; function to compute satd cost for 16 columns, 8 rows1294
1295
paddd xm6, xm71296
movd eax, xm61297
RET1298
+1299
+%macro SATD_HBD_AVX512_END 01300
+ vextracti32x8 ym7, m6, 11301
+ paddd ym6, ym71302
+ vextracti128 xm7, ym6, 11303
+ paddd xm6, xm71304
+ pxor xm7, xm71305
+ movhlps xm7, xm61306
+ paddd xm6, xm71307
+ pshufd xm7, xm6, 11308
+ paddd xm6, xm71309
+ movd eax, xm61310
+%endmacro1311
+%macro PROCESS_SATD_16x8_HBD_AVX512 0 ; function to compute satd cost for 16 columns, 8 rows1312
+ ; rows 0-31313
+ lea r6, [r0 + r1 * 4]1314
+ lea r7, [r2 + r3 * 4]1315
+ movu ym0, [r0]1316
+ movu ym4, [r2]1317
+ vinserti32x8 m0, [r6], 11318
+ vinserti32x8 m4, [r7], 11319
+ psubw m0, m41320
+ movu ym1, [r0 + r1]1321
+ movu ym5, [r2 + r3]1322
+ vinserti32x8 m1, [r6 + r1], 11323
+ vinserti32x8 m5, [r7 + r3], 11324
+ psubw m1, m51325
+ movu ym2, [r0 + r1 * 2]1326
+ movu ym4, [r2 + r3 * 2]1327
+ vinserti32x8 m2, [r6 + r1 * 2], 11328
+ vinserti32x8 m4, [r7 + r3 * 2], 11329
+ psubw m2, m41330
+ movu ym3, [r0 + r4]1331
+ movu ym5, [r2 + r5]1332
+ vinserti32x8 m3, [r6 + r4], 11333
+ vinserti32x8 m5, [r7 + r5], 11334
+ psubw m3, m51335
+1336
+ paddw m4, m0, m11337
+ psubw m1, m01338
+ paddw m0, m2, m31339
+ psubw m3, m21340
+ punpckhwd m2, m4, m11341
+ punpcklwd m4, m11342
+ punpckhwd m1, m0, m31343
+ punpcklwd m0, m31344
+ paddw m3, m4, m01345
+ psubw m0, m41346
+ paddw m4, m2, m11347
+ psubw m1, m21348
+ punpckhdq m2, m3, m01349
+ punpckldq m3, m01350
+ paddw m0, m3, m21351
+ psubw m2, m31352
+ punpckhdq m3, m4, m11353
+ punpckldq m4, m11354
+ paddw m1, m4, m31355
+ psubw m3, m41356
+ punpckhqdq m4, m0, m11357
+ punpcklqdq m0, m11358
+ pabsw m0, m01359
+ pabsw m4, m41360
+ pmaxsw m0, m0, m41361
+ punpckhqdq m1, m2, m31362
+ punpcklqdq m2, m31363
+ pabsw m2, m21364
+ pabsw m1, m11365
+ pmaxsw m2, m11366
+ pxor m7, m71367
+ mova m1, m01368
+ punpcklwd m1, m71369
+ paddd m6, m11370
+ mova m1, m01371
+ punpckhwd m1, m71372
+ paddd m6, m11373
+ pxor m7, m71374
+ mova m1, m21375
+ punpcklwd m1, m71376
+ paddd m6, m11377
+ mova m1, m21378
+ punpckhwd m1, m71379
+ paddd m6, m11380
+%endmacro1381
+%macro PROCESS_SATD_32x4_HBD_AVX512 0 ; function to compute satd cost for 32 columns, 4 rows1382
+ ; rows 0-31383
+ movu m0, [r0]1384
+ movu m4, [r2]1385
+ psubw m0, m41386
+ movu m1, [r0 + r1]1387
+ movu m5, [r2 + r3]1388
+ psubw m1, m51389
+ movu m2, [r0 + r1 * 2]1390
+ movu m4, [r2 + r3 * 2]1391
+ psubw m2, m41392
+ movu m3, [r0 + r4]1393
+ movu m5, [r2 + r5]1394
+ psubw m3, m51395
+ paddw m4, m0, m11396
+ psubw m1, m01397
+ paddw m0, m2, m31398
+ psubw m3, m21399
+ punpckhwd m2, m4, m11400
+ punpcklwd m4, m11401
+ punpckhwd m1, m0, m31402
+ punpcklwd m0, m31403
+ paddw m3, m4, m01404
+ psubw m0, m41405
+ paddw m4, m2, m11406
+ psubw m1, m21407
+ punpckhdq m2, m3, m01408
+ punpckldq m3, m01409
+ paddw m0, m3, m21410
+ psubw m2, m31411
+ punpckhdq m3, m4, m11412
+ punpckldq m4, m11413
+ paddw m1, m4, m31414
+ psubw m3, m41415
+ punpckhqdq m4, m0, m11416
+ punpcklqdq m0, m11417
+ pabsw m0, m01418
+ pabsw m4, m41419
+ pmaxsw m0, m0, m41420
+ punpckhqdq m1, m2, m31421
+ punpcklqdq m2, m31422
+ pabsw m2, m21423
+ pabsw m1, m11424
+ pmaxsw m2, m11425
+ pxor m7, m71426
+ mova m1, m01427
+ punpcklwd m1, m71428
+ paddd m6, m11429
+ mova m1, m01430
+ punpckhwd m1, m71431
+ paddd m6, m11432
+ pxor m7, m71433
+ mova m1, m21434
+ punpcklwd m1, m71435
+ paddd m6, m11436
+ mova m1, m21437
+ punpckhwd m1, m71438
+ paddd m6, m11439
+%endmacro1440
+1441
+%macro SATD_16xN_HBD_AVX512 11442
+INIT_ZMM avx5121443
+cglobal pixel_satd_16x%1, 4,8,81444
+ add r1d, r1d1445
+ add r3d, r3d1446
+ lea r4, [3 * r1]1447
+ lea r5, [3 * r3]1448
+ pxor m6, m61449
+1450
+%rep %1/8 - 11451
+ PROCESS_SATD_16x8_HBD_AVX5121452
+ lea r0, [r6 + 4 * r1]1453
+ lea r2, [r7 + 4 * r3]1454
+%endrep1455
+ PROCESS_SATD_16x8_HBD_AVX5121456
+ SATD_HBD_AVX512_END1457
+ RET1458
+%endmacro1459
+1460
+SATD_16xN_HBD_AVX512 81461
+SATD_16xN_HBD_AVX512 161462
+SATD_16xN_HBD_AVX512 321463
+SATD_16xN_HBD_AVX512 641464
+1465
+%macro SATD_32xN_HBD_AVX512 11466
+INIT_ZMM avx5121467
+cglobal pixel_satd_32x%1, 4,8,81468
+ add r1d, r1d1469
+ add r3d, r3d1470
+ lea r4, [3 * r1]1471
+ lea r5, [3 * r3]1472
+ pxor m6, m61473
+ mov r6, r01474
+ mov r7, r21475
+%rep %1/4 - 11476
+ PROCESS_SATD_32x4_HBD_AVX5121477
+ lea r0, [r0 + 4 * r1]1478
+ lea r2, [r2 + 4 * r3]1479
+%endrep1480
+ PROCESS_SATD_32x4_HBD_AVX5121481
+ SATD_HBD_AVX512_END1482
+ RET1483
+%endmacro1484
+1485
+SATD_32xN_HBD_AVX512 81486
+SATD_32xN_HBD_AVX512 161487
+SATD_32xN_HBD_AVX512 241488
+SATD_32xN_HBD_AVX512 321489
+SATD_32xN_HBD_AVX512 641490
+INIT_ZMM avx5121491
+cglobal pixel_satd_48x64, 4,10,81492
+ add r1d, r1d1493
+ add r3d, r3d1494
+ lea r4, [3 * r1]1495
+ lea r5, [3 * r3]1496
+ pxor m6, m61497
+ mov r8, r01498
+ mov r9, r21499
+1500
+%rep 151501
+ PROCESS_SATD_32x4_HBD_AVX5121502
+ lea r0, [r0 + 4 * r1]1503
+ lea r2, [r2 + 4 * r3]1504
+%endrep1505
+ PROCESS_SATD_32x4_HBD_AVX5121506
+ lea r0, [r8 + mmsize]1507
+ lea r2, [r9 + mmsize]1508
+%rep 71509
+ PROCESS_SATD_16x8_HBD_AVX5121510
+ lea r0, [r6 + 4 * r1]1511
+ lea r2, [r7 + 4 * r3]1512
+%endrep1513
+ PROCESS_SATD_16x8_HBD_AVX5121514
+ SATD_HBD_AVX512_END1515
+ RET1516
+1517
+%macro SATD_64xN_HBD_AVX512 11518
+INIT_ZMM avx5121519
+cglobal pixel_satd_64x%1, 4,8,81520
+ add r1d, r1d1521
+ add r3d, r3d1522
+ lea r4, [3 * r1]1523
+ lea r5, [3 * r3]1524
+ pxor m6, m61525
+ mov r6, r01526
+ mov r7, r21527
+%rep %1/4 - 11528
+ PROCESS_SATD_32x4_HBD_AVX5121529
+ lea r0, [r0 + 4 * r1]1530
+ lea r2, [r2 + 4 * r3]1531
+%endrep1532
+ PROCESS_SATD_32x4_HBD_AVX5121533
+ lea r0, [r6 + mmsize]1534
+ lea r2, [r7 + mmsize]1535
+%rep %1/4 - 11536
+ PROCESS_SATD_32x4_HBD_AVX5121537
+ lea r0, [r0 + 4 * r1]1538
+ lea r2, [r2 + 4 * r3]1539
+%endrep1540
+ PROCESS_SATD_32x4_HBD_AVX5121541
+ SATD_HBD_AVX512_END1542
+ RET1543
+%endmacro1544
+1545
+SATD_64xN_HBD_AVX512 161546
+SATD_64xN_HBD_AVX512 321547
+SATD_64xN_HBD_AVX512 481548
+SATD_64xN_HBD_AVX512 641549
%endif ; ARCH_X86_64 == 1 && HIGH_BIT_DEPTH == 11550
1551
1552
1553
;lea %8, [%8+4*r3]1554
%endmacro1555
1556
+%if ARCH_X86_641557
INIT_YMM avx21558
cglobal pixel_satd_8x8, 4,4,71559
1560
1561
1562
movd eax, xm01563
RET1564
-1565
+%endif1566
%endif ; HIGH_BIT_DEPTH == 1 && BIT_DEPTH == 101567
x265_2.7.tar.gz/source/common/x86/pixel-util.h -> x265_2.9.tar.gz/source/common/x86/pixel-util.h
Changed
33
1
2
3
#define DEFINE_UTILS(cpu) \4
FUNCDEF_TU_S2(void, getResidual, cpu, const pixel* fenc, const pixel* pred, int16_t* residual, intptr_t stride); \5
+ FUNCDEF_TU_S2(void, getResidual_aligned, cpu, const pixel* fenc, const pixel* pred, int16_t* residual, intptr_t stride); \6
FUNCDEF_TU_S2(void, transpose, cpu, pixel* dest, const pixel* src, intptr_t stride); \7
FUNCDEF_TU(int, count_nonzero, cpu, const int16_t* quantCoeff); \8
uint32_t PFX(quant_ ## cpu(const int16_t* coef, const int32_t* quantCoeff, int32_t* deltaU, int16_t* qCoef, int qBits, int add, int numCoeff)); \9
10
void PFX(weight_pp_ ## cpu(const pixel* src, pixel* dst, intptr_t stride, int width, int height, int w0, int round, int shift, int offset)); \11
void PFX(weight_sp_ ## cpu(const int16_t* src, pixel* dst, intptr_t srcStride, intptr_t dstStride, int width, int height, int w0, int round, int shift, int offset)); \12
void PFX(scale1D_128to64_ ## cpu(pixel*, const pixel*)); \13
+ void PFX(scale1D_128to64_aligned_ ## cpu(pixel*, const pixel*)); \14
void PFX(scale2D_64to32_ ## cpu(pixel*, const pixel*, intptr_t)); \15
uint32_t PFX(costCoeffRemain_ ## cpu(uint16_t *absCoeff, int numNonZero, int idx)); \16
uint32_t PFX(costC1C2Flag_sse2(uint16_t *absCoeff, intptr_t numNonZero, uint8_t *baseCtxMod, intptr_t ctxOffset)); \17
18
DEFINE_UTILS(ssse3);19
DEFINE_UTILS(sse4);20
DEFINE_UTILS(avx2);21
+DEFINE_UTILS(avx512);22
23
#undef DEFINE_UTILS24
25
26
uint32_t PFX(costCoeffNxN_sse4(const uint16_t *scan, const coeff_t *coeff, intptr_t trSize, uint16_t *absCoeff, const uint8_t *tabSigCtx, uint32_t scanFlagMask, uint8_t *baseCtx, int offset, int scanPosSigOff, int subPosBase));27
uint32_t PFX(costCoeffNxN_avx2_bmi2(const uint16_t *scan, const coeff_t *coeff, intptr_t trSize, uint16_t *absCoeff, const uint8_t *tabSigCtx, uint32_t scanFlagMask, uint8_t *baseCtx, int offset, int scanPosSigOff, int subPosBase));28
29
+int PFX(count_nonzero_16x16_avx512(const int16_t* quantCoeff));30
+int PFX(count_nonzero_32x32_avx512(const int16_t* quantCoeff));31
+32
#endif // ifndef X265_PIXEL_UTIL_H33
x265_2.7.tar.gz/source/common/x86/pixel-util8.asm -> x265_2.9.tar.gz/source/common/x86/pixel-util8.asm
Changed
1798
1
2
;* Authors: Min Chen <chenm003@163.com> <min.chen@multicorewareinc.com>3
;* Nabajit Deka <nabajit@multicorewareinc.com>4
;* Rajesh Paulraj <rajesh@multicorewareinc.com>5
+;* Praveen Kumar Tiwari <praveen@multicorewareinc.com>6
;*7
;* This program is free software; you can redistribute it and/or modify8
;* it under the terms of the GNU General Public License as published by9
10
%include "x86inc.asm"11
%include "x86util.asm"12
13
-SECTION_RODATA 3214
+SECTION_RODATA 6415
+16
+var_shuf_avx512: db 0,-1, 1,-1, 2,-1, 3,-1, 4,-1, 5,-1, 6,-1, 7,-117
+ db 8,-1, 9,-1,10,-1,11,-1,12,-1,13,-1,14,-1,15,-118
+ALIGN 6419
+const dequant_shuf1_avx512, dq 0, 2, 4, 6, 1, 3, 5, 720
+const dequant_shuf2_avx512, dq 0, 4, 1, 5, 2, 6, 3, 721
22
%if BIT_DEPTH == 1223
ssim_c1: times 4 dd 107321.76 ; .01*.01*4095*4095*6424
25
%endrep26
RET27
%endif28
+29
+%macro PROCESS_GETRESIDUAL32_W4_HBD_AVX512 030
+ movu m0, [r0]31
+ movu m1, [r0 + r3]32
+ movu m2, [r0 + r3 * 2]33
+ movu m3, [r0 + r4]34
+ lea r0, [r0 + r3 * 4]35
+36
+ movu m4, [r1]37
+ movu m5, [r1 + r3]38
+ movu m6, [r1 + r3 * 2]39
+ movu m7, [r1 + r4]40
+ lea r1, [r1 + r3 * 4]41
+42
+ psubw m0, m443
+ psubw m1, m544
+ psubw m2, m645
+ psubw m3, m746
+47
+ movu [r2], m048
+ movu [r2 + r3], m149
+ movu [r2 + r3 * 2], m250
+ movu [r2 + r4], m351
+ lea r2, [r2 + r3 * 4]52
+%endmacro53
+54
+%macro PROCESS_GETRESIDUAL32_W4_HBD_AVX512_END 055
+ movu m0, [r0]56
+ movu m1, [r0 + r3]57
+ movu m2, [r0 + r3 * 2]58
+ movu m3, [r0 + r4]59
+60
+ movu m4, [r1]61
+ movu m5, [r1 + r3]62
+ movu m6, [r1 + r3 * 2]63
+ movu m7, [r1 + r4]64
+65
+ psubw m0, m466
+ psubw m1, m567
+ psubw m2, m668
+ psubw m3, m769
+70
+ movu [r2], m071
+ movu [r2 + r3], m172
+ movu [r2 + r3 * 2], m273
+ movu [r2 + r4], m374
+%endmacro75
+76
+%macro PROCESS_GETRESIDUAL32_W4_AVX512 077
+ pmovzxbw m0, [r0]78
+ pmovzxbw m1, [r0 + r3]79
+ pmovzxbw m2, [r0 + r3 * 2]80
+ pmovzxbw m3, [r0 + r4]81
+ lea r0, [r0 + r3 * 4]82
+83
+ pmovzxbw m4, [r1]84
+ pmovzxbw m5, [r1 + r3]85
+ pmovzxbw m6, [r1 + r3 * 2]86
+ pmovzxbw m7, [r1 + r4]87
+ lea r1, [r1 + r3 * 4]88
+89
+ psubw m0, m490
+ psubw m1, m591
+ psubw m2, m692
+ psubw m3, m793
+94
+ movu [r2], m095
+ movu [r2 + r3 * 2], m196
+ lea r2, [r2 + r3 * 4]97
+ movu [r2], m298
+ movu [r2 + r3 * 2], m399
+ lea r2, [r2 + r3 * 4]100
+%endmacro101
+102
+%macro PROCESS_GETRESIDUAL32_W4_AVX512_END 0103
+ pmovzxbw m0, [r0]104
+ pmovzxbw m1, [r0 + r3]105
+ pmovzxbw m2, [r0 + r3 * 2]106
+ pmovzxbw m3, [r0 + r4]107
+108
+ pmovzxbw m4, [r1]109
+ pmovzxbw m5, [r1 + r3]110
+ pmovzxbw m6, [r1 + r3 * 2]111
+ pmovzxbw m7, [r1 + r4]112
+113
+ psubw m0, m4114
+ psubw m1, m5115
+ psubw m2, m6116
+ psubw m3, m7117
+118
+ movu [r2], m0119
+ movu [r2 + r3 * 2], m1120
+ lea r2, [r2 + r3 * 4]121
+ movu [r2], m2122
+ movu [r2 + r3 * 2], m3123
+%endmacro124
+125
+126
+%if HIGH_BIT_DEPTH127
+INIT_ZMM avx512128
+cglobal getResidual32, 4,5,8129
+ add r3, r3130
+ lea r4, [r3 * 3]131
+132
+ PROCESS_GETRESIDUAL32_W4_HBD_AVX512133
+ PROCESS_GETRESIDUAL32_W4_HBD_AVX512134
+ PROCESS_GETRESIDUAL32_W4_HBD_AVX512135
+ PROCESS_GETRESIDUAL32_W4_HBD_AVX512136
+ PROCESS_GETRESIDUAL32_W4_HBD_AVX512137
+ PROCESS_GETRESIDUAL32_W4_HBD_AVX512138
+ PROCESS_GETRESIDUAL32_W4_HBD_AVX512139
+ PROCESS_GETRESIDUAL32_W4_HBD_AVX512_END140
+ RET141
+%else142
+INIT_ZMM avx512143
+cglobal getResidual32, 4,5,8144
+ lea r4, [r3 * 3]145
+146
+ PROCESS_GETRESIDUAL32_W4_AVX512147
+ PROCESS_GETRESIDUAL32_W4_AVX512148
+ PROCESS_GETRESIDUAL32_W4_AVX512149
+ PROCESS_GETRESIDUAL32_W4_AVX512150
+ PROCESS_GETRESIDUAL32_W4_AVX512151
+ PROCESS_GETRESIDUAL32_W4_AVX512152
+ PROCESS_GETRESIDUAL32_W4_AVX512153
+ PROCESS_GETRESIDUAL32_W4_AVX512_END154
+ RET155
+%endif156
+157
+%macro PROCESS_GETRESIDUAL32_ALIGNED_W4_HBD_AVX512 0158
+ movu m0, [r0]159
+ movu m1, [r0 + r3]160
+ movu m2, [r0 + r3 * 2]161
+ movu m3, [r0 + r4]162
+ lea r0, [r0 + r3 * 4]163
+164
+ movu m4, [r1]165
+ movu m5, [r1 + r3]166
+ movu m6, [r1 + r3 * 2]167
+ movu m7, [r1 + r4]168
+ lea r1, [r1 + r3 * 4]169
+170
+ psubw m0, m4171
+ psubw m1, m5172
+ psubw m2, m6173
+ psubw m3, m7174
+175
+ movu [r2], m0176
+ movu [r2 + r3], m1177
+ movu [r2 + r3 * 2], m2178
+ movu [r2 + r4], m3179
+ lea r2, [r2 + r3 * 4]180
+%endmacro181
+182
+%macro PROCESS_GETRESIDUAL32_ALIGNED_W4_HBD_AVX512_END 0183
+ movu m0, [r0]184
+ movu m1, [r0 + r3]185
+ movu m2, [r0 + r3 * 2]186
+ movu m3, [r0 + r4]187
+188
+ movu m4, [r1]189
+ movu m5, [r1 + r3]190
+ movu m6, [r1 + r3 * 2]191
+ movu m7, [r1 + r4]192
+193
+ psubw m0, m4194
+ psubw m1, m5195
+ psubw m2, m6196
+ psubw m3, m7197
+198
+ movu [r2], m0199
+ movu [r2 + r3], m1200
+ movu [r2 + r3 * 2], m2201
+ movu [r2 + r4], m3202
+%endmacro203
+204
+%macro PROCESS_GETRESIDUAL32_ALIGNED_W4_AVX512 0205
+ pmovzxbw m0, [r0]206
+ pmovzxbw m1, [r0 + r3]207
+ pmovzxbw m2, [r0 + r3 * 2]208
+ pmovzxbw m3, [r0 + r4]209
+ lea r0, [r0 + r3 * 4]210
+211
+ pmovzxbw m4, [r1]212
+ pmovzxbw m5, [r1 + r3]213
+ pmovzxbw m6, [r1 + r3 * 2]214
+ pmovzxbw m7, [r1 + r4]215
+ lea r1, [r1 + r3 * 4]216
+217
+ psubw m0, m4218
+ psubw m1, m5219
+ psubw m2, m6220
+ psubw m3, m7221
+222
+ movu [r2], m0223
+ movu [r2 + r3 * 2], m1224
+ lea r2, [r2 + r3 * 4]225
+ movu [r2], m2226
+ movu [r2 + r3 * 2], m3227
+ lea r2, [r2 + r3 * 4]228
+%endmacro229
+230
+%macro PROCESS_GETRESIDUAL32_ALIGNED_W4_AVX512_END 0231
+ pmovzxbw m0, [r0]232
+ pmovzxbw m1, [r0 + r3]233
+ pmovzxbw m2, [r0 + r3 * 2]234
+ pmovzxbw m3, [r0 + r4]235
+236
+ pmovzxbw m4, [r1]237
+ pmovzxbw m5, [r1 + r3]238
+ pmovzxbw m6, [r1 + r3 * 2]239
+ pmovzxbw m7, [r1 + r4]240
+241
+ psubw m0, m4242
+ psubw m1, m5243
+ psubw m2, m6244
+ psubw m3, m7245
+246
+ movu [r2], m0247
+ movu [r2 + r3 * 2], m1248
+ lea r2, [r2 + r3 * 4]249
+ movu [r2], m2250
+ movu [r2 + r3 * 2], m3251
+%endmacro252
+253
+254
+%if HIGH_BIT_DEPTH255
+INIT_ZMM avx512256
+cglobal getResidual_aligned32, 4,5,8257
+ add r3, r3258
+ lea r4, [r3 * 3]259
+260
+ PROCESS_GETRESIDUAL32_ALIGNED_W4_HBD_AVX512261
+ PROCESS_GETRESIDUAL32_ALIGNED_W4_HBD_AVX512262
+ PROCESS_GETRESIDUAL32_ALIGNED_W4_HBD_AVX512263
+ PROCESS_GETRESIDUAL32_ALIGNED_W4_HBD_AVX512264
+ PROCESS_GETRESIDUAL32_ALIGNED_W4_HBD_AVX512265
+ PROCESS_GETRESIDUAL32_ALIGNED_W4_HBD_AVX512266
+ PROCESS_GETRESIDUAL32_ALIGNED_W4_HBD_AVX512267
+ PROCESS_GETRESIDUAL32_ALIGNED_W4_HBD_AVX512_END268
+ RET269
+%else270
+INIT_ZMM avx512271
+cglobal getResidual_aligned32, 4,5,8272
+ lea r4, [r3 * 3]273
+274
+ PROCESS_GETRESIDUAL32_ALIGNED_W4_AVX512275
+ PROCESS_GETRESIDUAL32_ALIGNED_W4_AVX512276
+ PROCESS_GETRESIDUAL32_ALIGNED_W4_AVX512277
+ PROCESS_GETRESIDUAL32_ALIGNED_W4_AVX512278
+ PROCESS_GETRESIDUAL32_ALIGNED_W4_AVX512279
+ PROCESS_GETRESIDUAL32_ALIGNED_W4_AVX512280
+ PROCESS_GETRESIDUAL32_ALIGNED_W4_AVX512281
+ PROCESS_GETRESIDUAL32_ALIGNED_W4_AVX512_END282
+ RET283
+%endif284
;-----------------------------------------------------------------------------285
; uint32_t quant(int16_t *coef, int32_t *quantCoeff, int32_t *deltaU, int16_t *qCoef, int qBits, int add, int numCoeff);286
;-----------------------------------------------------------------------------287
288
%endif ; ARCH_X86_64 == 1289
290
291
+%if ARCH_X86_64 == 1292
+INIT_ZMM avx512293
+cglobal quant, 5, 6, 22294
+ ; fill qbits295
+ movd xm4, r4d ; m4 = qbits296
+297
+ ; fill qbits-8298
+ sub r4d, 8299
+ movd xm6, r4d ; m6 = qbits8300
+301
+ ; fill offset302
+%if UNIX64 == 0303
+ vpbroadcastd m5, r5m ; m5 = add304
+%else ; Mac305
+ movd xm5, r5m306
+ vpbroadcastd m5, xm5 ; m5 = add307
+%endif308
+309
+ vbroadcasti32x8 m9, [pw_1]310
+311
+ mov r4d, r6m312
+ pxor m7, m7313
+ sub r4d, 32314
+ jl .coeff16315
+ add r4d, 32316
+ shr r4d, 5317
+ jmp .loop318
+319
+.coeff16:320
+ ; 16 coeff321
+ pxor m7, m7322
+ pmovsxwd m16, [r0] ; m16 = level323
+ pabsd m1, m16324
+ pmulld m1, [r1]325
+ paddd m17, m1, m5326
+ psrad m17, xm4 ; m17 = level1327
+328
+ pslld m3, m17, 8329
+ psrad m1, xm6330
+ psubd m1, m3 ; m1 = deltaU1331
+ movu [r2], m1332
+ vextracti64x4 ym19, m17, 1333
+ vextracti64x4 ym20, m16, 1334
+ psignd ym17, ym16335
+ psignd ym19, ym20336
+ packssdw ym17, ym19337
+ vpermq ym17, ym17, q3120338
+ movu [r3], ym17339
+340
+ pminuw ym17, ym9341
+ paddw ym7, ym17342
+343
+ ; sum count344
+ xorpd m0, m0345
+ psadbw ym7, ym0346
+ vextracti128 xm1, ym7, 1347
+ paddd xm7, xm1348
+ movhlps xm0, xm7349
+ paddd xm7, xm0350
+ movd eax, xm7351
+ RET352
+353
+.loop:354
+ ; 16 coeff355
+ pmovsxwd m16, [r0] ; m16 = level356
+ pabsd m1, m16357
+ pmulld m1, [r1]358
+ paddd m17, m1, m5359
+ psrad m17, xm4 ; m17 = level1360
+361
+ pslld m3, m17, 8362
+ psrad m1, xm6363
+ psubd m1, m3 ; m1 = deltaU1364
+ movu [r2], m1365
+ vextracti64x4 ym19, m17, 1366
+ vextracti64x4 ym20, m16, 1367
+ psignd ym17, ym16368
+ psignd ym19, ym20369
+ packssdw ym17, ym19370
+371
+ ; 16 coeff372
+ pmovsxwd m16, [r0 + mmsize/2] ; m16 = level373
+ pabsd m1, m16374
+ pmulld m1, [r1 + mmsize]375
+ paddd m18, m1, m5376
+ psrad m18, xm4 ; m2 = level1377
+378
+ pslld m8, m18, 8379
+ psrad m1, xm6380
+ psubd m1, m8 ; m1 = deltaU1381
+ movu [r2 + mmsize], m1382
+ vextracti64x4 ym21, m18, 1383
+ vextracti64x4 ym20, m16, 1384
+ psignd ym18, ym16385
+ psignd ym21, ym20386
+ packssdw ym18, ym21387
+ vinserti64x4 m17, m17, ym18, 1388
+ vpermq m17, m17, q3120389
+390
+ movu [r3], m17391
+392
+ pminuw m17, m9393
+ paddw m7, m17394
+395
+ add r0, mmsize396
+ add r1, mmsize * 2397
+ add r2, mmsize * 2398
+ add r3, mmsize399
+400
+ dec r4d401
+ jnz .loop402
+403
+ ; sum count404
+ xorpd m0, m0405
+ psadbw m7, m0406
+ vextracti32x8 ym1, m7, 1407
+ paddd ym7, ym1408
+ vextracti64x2 xm1, m7, 1409
+ paddd xm7, xm1410
+ pshufd xm1, xm7, 2411
+ paddd xm7, xm1412
+ movd eax, xm7413
+ RET414
+%endif ; ARCH_X86_64 == 1415
+416
+417
+418
;-----------------------------------------------------------------------------419
; uint32_t nquant(int16_t *coef, int32_t *quantCoeff, int16_t *qCoef, int qBits, int add, int numCoeff);420
;-----------------------------------------------------------------------------421
422
paddd xm5, xm0423
movd eax, xm5424
RET425
+%if ARCH_X86_64 == 1426
+INIT_ZMM avx512427
+cglobal nquant, 3,5,22428
+%if UNIX64 == 0429
+ vpbroadcastd m4, r4m430
+%else ; Mac431
+ movd xm4, r4m432
+ vpbroadcastd m4, xm4433
+%endif434
435
+ vbroadcasti32x8 m6, [pw_1]436
+ mov r4d, r5m437
+ pxor m5, m5438
+ movd xm3, r3m439
+ sub r4d, 16440
+ je .coeff16441
+ add r4d, 16442
+ shr r4d, 5443
+ jmp .loop444
+445
+.coeff16:446
+ pmovsxwd m16, [r0]447
+ pabsd m17, m16448
+ pmulld m17, [r1]449
+ paddd m17, m4450
+ psrad m17, xm3451
+452
+ vextracti64x4 ym19, m17, 1453
+ vextracti64x4 ym20, m16, 1454
+ psignd ym17, ym16455
+ psignd ym19, ym20456
+ packssdw ym17, ym19457
+ vpermq ym17, ym17, q3120458
+ pabsw ym17, ym17459
+ movu [r2], ym17460
+ pminuw ym17, ym6461
+ paddw ym5, ym17462
+ pxor m0, m0463
+ psadbw ym5, ym0464
+ vextracti128 xm0, ym5, 1465
+ paddd xm5, xm0466
+ pshufd xm0, xm5, 2467
+ paddd xm5, xm0468
+ movd eax, xm5469
+ RET470
+471
+.loop:472
+ pmovsxwd m16, [r0]473
+ pabsd m17, m16474
+ pmulld m17, [r1]475
+ paddd m17, m4476
+ psrad m17, xm3477
+ vextracti64x4 ym19, m17, 1478
+ vextracti64x4 ym20, m16, 1479
+ psignd ym17, ym16480
+ psignd ym19, ym20481
+ packssdw ym17, ym19482
+483
+ pmovsxwd m16, [r0 + mmsize/2]484
+ pabsd m18, m16485
+ pmulld m18, [r1 + mmsize]486
+ paddd m18, m4487
+ psrad m18, xm3488
+ vextracti64x4 ym21, m18, 1489
+ vextracti64x4 ym20, m16, 1490
+ psignd ym18, ym16491
+ psignd ym21, ym20492
+ packssdw ym18, ym21493
+ vinserti64x4 m17, m17, ym18, 1494
+ vpermq m17, m17, q3120495
+496
+ pabsw m17, m17497
+ movu [r2], m17498
+499
+ add r0, mmsize500
+ add r1, mmsize * 2501
+ add r2, mmsize502
+503
+ pminuw m17, m6504
+ paddw m5, m17505
+506
+ dec r4d507
+ jnz .loop508
+509
+ pxor m0, m0510
+ psadbw m5, m0511
+ vextracti32x8 ym1, m5, 1512
+ paddd ym5, ym1513
+ vextracti64x2 xm1, m5, 1514
+ paddd xm5, xm1515
+ pshufd xm1, xm5, 2516
+ paddd xm5, xm1517
+ movd eax, xm5518
+ RET519
+%endif ; ARCH_X86_64 == 1520
521
;-----------------------------------------------------------------------------522
; void dequant_normal(const int16_t* quantCoef, int32_t* coef, int num, int scale, int shift)523
524
jnz .loop525
RET526
527
+;----------------------------------------------------------------------------------------------------------------------528
+;void dequant_scaling(const int16_t* src, const int32_t* dequantCoef, int16_t* dst, int num, int mcqp_miper, int shift)529
+;----------------------------------------------------------------------------------------------------------------------530
+INIT_ZMM avx512531
+cglobal dequant_scaling, 6,7,8532
+ mova m6, [dequant_shuf1_avx512]533
+ mova m7, [dequant_shuf2_avx512]534
+ add r5d, 4535
+ mov r6d, r3d536
+ shr r3d, 5 ; num/32537
+ cmp r5d, r4d538
+ jle .skip539
+ sub r5d, r4d540
+ vpbroadcastd m0, [pd_1]541
+ movd xm1, r5d ; shift - per542
+ dec r5d543
+ movd xm2, r5d ; shift - per - 1544
+ pslld m0, xm2 ; 1 << shift - per - 1545
+546
+.part0:547
+ pmovsxwd m2, [r0]548
+ pmovsxwd m4, [r0 + 32]549
+ movu m3, [r1]550
+ movu m5, [r1 + 64]551
+ pmulld m2, m3552
+ pmulld m4, m5553
+ paddd m2, m0554
+ paddd m4, m0555
+ psrad m2, xm1556
+ psrad m4, xm1557
+ packssdw m2, m4558
+ vpermq m2, m6, m2559
+ cmp r6d, 16560
+ je .num16part0561
+ movu [r2], m2562
+563
+ add r0, 64564
+ add r1, 128565
+ add r2, 64566
+ dec r3d567
+ jnz .part0568
+ jmp .end569
+570
+.num16part0:571
+ movu [r2], ym2572
+ jmp .end573
+574
+.skip:575
+ sub r4d, r5d ; per - shift576
+ movd xm0, r4d577
+578
+.part1:579
+ pmovsxwd m2, [r0]580
+ pmovsxwd m4, [r0 + 32]581
+ movu m3, [r1]582
+ movu m5, [r1 + 64]583
+ pmulld m2, m3584
+ pmulld m4, m5585
+ packssdw m2, m4586
+587
+ vextracti32x8 ym4, m2, 1588
+ pmovsxwd m1, ym2589
+ pmovsxwd m2, ym4590
+ pslld m1, xm0591
+ pslld m2, xm0592
+ packssdw m1, m2593
+594
+ vpermq m1, m7, m1595
+ cmp r6d, 16596
+ je .num16part1597
+ movu [r2], m1598
+599
+ add r0, 64600
+ add r1, 128601
+ add r2, 64602
+ dec r3d603
+ jnz .part1604
+605
+.num16part1:606
+ movu [r2], ym1607
+608
+.end:609
+ RET610
+611
+INIT_ZMM avx512612
+cglobal dequant_normal, 5,5,7613
+ vpbroadcastd m2, [pw_1] ; m2 = word [1]614
+ vpbroadcastd m5, [pd_32767] ; m5 = dword [32767]615
+ vpbroadcastd m6, [pd_n32768] ; m6 = dword [-32768]616
+%if HIGH_BIT_DEPTH617
+ cmp r3d, 32767618
+ jle .skip619
+ shr r3d, (BIT_DEPTH - 8)620
+ sub r4d, (BIT_DEPTH - 8)621
+.skip:622
+%endif623
+ movd xm0, r4d ; m0 = shift624
+ add r4d, -1+16625
+ bts r3d, r4d626
+627
+ movd xm1, r3d628
+ vpbroadcastd m1, xm1 ; m1 = dword [add scale]629
+630
+ ; m0 = shift631
+ ; m1 = scale632
+ ; m2 = word [1]633
+ mov r3d, r2d634
+ shr r2d, 5635
+.loop:636
+ movu m3, [r0]637
+ punpckhwd m4, m3, m2638
+ punpcklwd m3, m2639
+ pmaddwd m3, m1 ; m3 = dword (clipQCoef * scale + add)640
+ pmaddwd m4, m1641
+ psrad m3, xm0642
+ psrad m4, xm0643
+ pminsd m3, m5644
+ pmaxsd m3, m6645
+ pminsd m4, m5646
+ pmaxsd m4, m6647
+ packssdw m3, m4648
+649
+ mova [r1 + 0 * mmsize/2], ym3650
+ cmp r3d, 16651
+ je .num16652
+ vextracti32x8 [r1 + 1 * mmsize/2], m3, 1653
+654
+ add r0, mmsize655
+ add r1, mmsize656
+657
+ dec r2d658
+ jnz .loop659
+ RET660
+.num16:661
+ RET662
+663
664
;-----------------------------------------------------------------------------665
; int x265_count_nonzero_4x4_sse2(const int16_t *quantCoeff);666
667
movd eax, xm0668
RET669
670
+;-----------------------------------------------------------------------------671
+; int x265_count_nonzero_16x16_avx512(const int16_t *quantCoeff);672
+;-----------------------------------------------------------------------------673
+%if ARCH_X86_64674
+INIT_ZMM avx512675
+cglobal count_nonzero_16x16, 1,4,2676
+ mov r1, 0xFFFFFFFFFFFFFFFF677
+ kmovq k2, r1678
+ xor r3, r3679
+ pxor m0, m0680
681
+%assign x 0682
+%rep 4683
+ movu m1, [r0 + x]684
+ vpacksswb m1, [r0 + x + 64]685
+%assign x x+128686
+ vpcmpb k1 {k2}, m1, m0, 00000100b687
+ kmovq r1, k1688
+ popcnt r2, r1689
+ add r3d, r2d690
+%endrep691
+ mov eax, r3d692
+ RET693
+%endif694
;-----------------------------------------------------------------------------695
; int x265_count_nonzero_32x32_sse2(const int16_t *quantCoeff);696
;-----------------------------------------------------------------------------697
698
RET699
700
701
+;-----------------------------------------------------------------------------702
+; int x265_count_nonzero_32x32_avx512(const int16_t *quantCoeff);703
+;-----------------------------------------------------------------------------704
+%if ARCH_X86_64705
+INIT_ZMM avx512706
+cglobal count_nonzero_32x32, 1,4,2707
+ mov r1, 0xFFFFFFFFFFFFFFFF708
+ kmovq k2, r1709
+ xor r3, r3710
+ pxor m0, m0711
+712
+%assign x 0713
+%rep 16714
+ movu m1, [r0 + x]715
+ vpacksswb m1, [r0 + x + 64]716
+%assign x x+128717
+ vpcmpb k1 {k2}, m1, m0, 00000100b718
+ kmovq r1, k1719
+ popcnt r2, r1720
+ add r3d, r2d721
+%endrep722
+ mov eax, r3d723
+ RET724
+%endif725
;-----------------------------------------------------------------------------------------------------------------------------------------------726
;void weight_pp(pixel *src, pixel *dst, intptr_t stride, int width, int height, int w0, int round, int shift, int offset)727
;-----------------------------------------------------------------------------------------------------------------------------------------------728
729
jnz .loopH730
RET731
%endif732
+733
+%if HIGH_BIT_DEPTH734
+INIT_ZMM avx512735
+cglobal weight_pp, 6, 7, 7736
+%define correction (14 - BIT_DEPTH)737
+ mov r6d, r6m738
+ shl r6d, 16 - correction739
+ or r6d, r5d740
+741
+ movd xm0, r6d742
+ vpbroadcastd m0, xm0743
+ mov r5d, r7m744
+ sub r5d, correction745
+ movd xm1, r5d746
+747
+ vpbroadcastd m2, r8m748
+ vbroadcasti32x8 m5, [pw_1]749
+ vbroadcasti32x8 m6, [pw_pixel_max]750
+751
+ add r2d, r2d752
+ add r3d, r3d753
+ sub r2d, r3d754
+ shr r3d, 6755
+756
+.loopH:757
+ mov r5d, r3d758
+759
+.loopW:760
+ movu m4, [r0]761
+ punpcklwd m3, m4, m5762
+ pmaddwd m3, m0763
+ psrad m3, xm1764
+ paddd m3, m2765
+766
+ punpckhwd m4, m5767
+ pmaddwd m4, m0768
+ psrad m4, xm1769
+ paddd m4, m2770
+771
+ packusdw m3, m4772
+ pminuw m3, m6773
+ movu [r1], m3774
+775
+ add r0, 64776
+ add r1, 64777
+778
+ dec r5d779
+ jnz .loopW780
+781
+ lea r0, [r0 + r2]782
+ lea r1, [r1 + r2]783
+784
+ dec r4d785
+ jnz .loopH786
+%undef correction787
+ RET788
+%else789
+INIT_ZMM avx512790
+cglobal weight_pp, 6, 7, 6791
+792
+ shl r5d, 6793
+ mov r6d, r6m794
+ shl r6d, 16795
+ or r6d, r5d796
+797
+ movd xm0, r6d798
+ vpbroadcastd m0, xm0799
+ movd xm1, r7m800
+ vpbroadcastd m2, r8m801
+802
+ vbroadcasti32x8 m5, [pw_1]803
+804
+ sub r2d, r3d805
+ shr r3d, 5806
+807
+.loopH:808
+ mov r5d, r3d809
+810
+.loopW:811
+ pmovzxbw m4, [r0]812
+ punpcklwd m3, m4, m5813
+ pmaddwd m3, m0814
+ psrad m3, xm1815
+ paddd m3, m2816
+817
+ punpckhwd m4, m5818
+ pmaddwd m4, m0819
+ psrad m4, xm1820
+ paddd m4, m2821
+822
+ packssdw m3, m4823
+ vextracti64x4 ym4, m3, 1824
+ packuswb ym3, ym4825
+ vpermq ym3, ym3, q3120826
+ movu [r1], ym3827
+828
+ add r0, 32829
+ add r1, 32830
+831
+ dec r5d832
+ jnz .loopW833
+834
+ lea r0, [r0 + r2]835
+ lea r1, [r1 + r2]836
+837
+ dec r4d838
+ jnz .loopH839
+ RET840
+%endif841
+842
;-------------------------------------------------------------------------------------------------------------------------------------------------843
;void weight_sp(int16_t *src, pixel *dst, intptr_t srcStride, intptr_t dstStride, int width, int height, int w0, int round, int shift, int offset)844
;-------------------------------------------------------------------------------------------------------------------------------------------------845
846
%endif847
%endif848
849
+%if ARCH_X86_64 == 1850
+%if HIGH_BIT_DEPTH851
+INIT_ZMM avx512852
+cglobal weight_sp, 6,9,8853
+ vbroadcasti32x8 m1, [pw_pixel_max]854
+ vbroadcasti32x8 m2, [pw_1]855
+856
+ mov r6d, r7m857
+ shl r6d, 16858
+ or r6d, r6m859
+ movd xm3, r6d860
+ vpbroadcastd m3, xm3 ; m3 = [round w0]861
+ movd xm4, r8m ; m4 = [shift]862
+ vpbroadcastd m5, r9m ; m5 = [offset]863
+864
+ ; correct row stride865
+ add r3d, r3d866
+ add r2d, r2d867
+ mov r6d, r4d868
+ and r6d, ~(mmsize / SIZEOF_PIXEL - 1)869
+ shl r6d, 1870
+ sub r3d, r6d871
+ sub r2d, r6d872
+873
+ mov r6d, r4d874
+ and r6d, (mmsize / SIZEOF_PIXEL - 1)875
+876
+.loopH:877
+ mov r6d, r4d878
+879
+.loopW:880
+ movu m6, [r0]881
+ vbroadcasti32x8 m8, [pw_2000]882
+ paddw m6, m8883
+884
+ punpcklwd m7, m6, m2885
+ pmaddwd m7, m3 ;(round w0)886
+ psrad m7, xm4 ;(shift)887
+ paddd m7, m5 ;(offset)888
+889
+ punpckhwd m6, m2890
+ pmaddwd m6, m3891
+ psrad m6, xm4892
+ paddd m6, m5893
+894
+ packusdw m7, m6895
+ pminuw m7, m1896
+897
+ sub r6d, (mmsize / SIZEOF_PIXEL)898
+ jl .widthLess30899
+ movu [r1], m7900
+ lea r0, [r0 + mmsize]901
+ lea r1, [r1 + mmsize]902
+ je .nextH903
+ jmp .loopW904
+905
+.widthLess30:906
+ mov r8d, 0xFFFFFFFF907
+ NEG r6d908
+ shrx r8d, r8d, r6d909
+ kmovd k1, r8d910
+ vmovdqu16 [r1] {k1}, m7911
+ jmp .nextH912
+913
+.nextH:914
+ add r0, r2915
+ add r1, r3916
+917
+ dec r5d918
+ jnz .loopH919
+ RET920
+921
+%else922
+INIT_ZMM avx512923
+cglobal weight_sp, 6, 10, 7924
+ mov r7d, r7m925
+ shl r7d, 16926
+ or r7d, r6m927
+ movd xm0, r7d928
+ vpbroadcastd m0, xm0 ; m0 = times 8 dw w0, round929
+ movd xm1, r8m ; m1 = [shift]930
+ vpbroadcastd m2, r9m ; m2 = times 16 dw offset931
+ vpbroadcastw m3, [pw_1]932
+ vpbroadcastw m4, [pw_2000]933
+934
+ add r2d, r2d ; 2 * srcstride935
+936
+ mov r7, r0937
+ mov r8, r1938
+.loopH:939
+ mov r6d, r4d ; width940
+941
+ ; save old src and dst942
+ mov r0, r7 ; src943
+ mov r1, r8 ; dst944
+945
+.loopW:946
+ movu m5, [r0]947
+ paddw m5, m4948
+949
+ punpcklwd m6, m5, m3950
+ pmaddwd m6, m0951
+ psrad m6, xm1952
+ paddd m6, m2953
+954
+ punpckhwd m5, m3955
+ pmaddwd m5, m0956
+ psrad m5, xm1957
+ paddd m5, m2958
+959
+ packssdw m6, m5960
+ vextracti64x4 ym5, m6, 1961
+ packuswb ym6, ym5962
+ vpermq ym6, ym6, q3120963
+964
+ sub r6d, 32965
+ jl .widthLess30966
+ movu [r1], ym6967
+ je .nextH968
+ add r0, 64969
+ add r1, 32970
+ jmp .loopW971
+972
+973
+.widthLess30:974
+ mov r9d, 0xFFFFFFFF975
+ NEG r6d976
+ shrx r9d, r9d, r6d977
+ kmovd k1, r9d978
+ vmovdqu8 [r1] {k1}, ym6979
+ jmp .nextH980
+981
+.nextH:982
+ lea r7, [r7 + r2]983
+ lea r8, [r8 + r3]984
+985
+ dec r5d986
+ jnz .loopH987
+ RET988
+%endif989
+%endif990
+991
+992
;-----------------------------------------------------------------993
; void transpose_4x4(pixel *dst, pixel *src, intptr_t stride)994
;-----------------------------------------------------------------995
996
RET997
%endif998
999
+%if HIGH_BIT_DEPTH == 01000
+INIT_ZMM avx5121001
+cglobal scale1D_128to64, 2, 2, 71002
+ pxor m4, m41003
+ mova m6, [dequant_shuf1_avx512]1004
+ vbroadcasti32x8 m5, [pb_1]1005
+1006
+ ;Top pixel1007
+ movu m0, [r1]1008
+ movu m1, [r1 + 1 * mmsize]1009
+ movu m2, [r1 + 2 * mmsize]1010
+ movu m3, [r1 + 3 * mmsize]1011
+1012
+ pmaddubsw m0, m51013
+ pavgw m0, m41014
+ pmaddubsw m1, m51015
+ pavgw m1, m41016
+ packuswb m0, m11017
+ vpermq m0, m6, m01018
+ movu [r0], m01019
+1020
+ ;Left pixel1021
+ pmaddubsw m2, m51022
+ pavgw m2, m41023
+ pmaddubsw m3, m51024
+ pavgw m3, m41025
+ packuswb m2, m31026
+ vpermq m2, m6, m21027
+ movu [r0 + mmsize], m21028
+ RET1029
+1030
+INIT_ZMM avx5121031
+cglobal scale1D_128to64_aligned, 2, 2, 71032
+ pxor m4, m41033
+ mova m6, [dequant_shuf1_avx512]1034
+ vbroadcasti32x8 m5, [pb_1]1035
+1036
+ ;Top pixel1037
+ mova m0, [r1]1038
+ mova m1, [r1 + 1 * mmsize]1039
+ mova m2, [r1 + 2 * mmsize]1040
+ mova m3, [r1 + 3 * mmsize]1041
+1042
+ pmaddubsw m0, m51043
+ pavgw m0, m41044
+ pmaddubsw m1, m51045
+ pavgw m1, m41046
+ packuswb m0, m11047
+ vpermq m0, m6, m01048
+ mova [r0], m01049
+1050
+ ;Left pixel1051
+ pmaddubsw m2, m51052
+ pavgw m2, m41053
+ pmaddubsw m3, m51054
+ pavgw m3, m41055
+ packuswb m2, m31056
+ vpermq m2, m6, m21057
+ mova [r0 + mmsize], m21058
+ RET1059
+%endif1060
+1061
;-----------------------------------------------------------------1062
; void scale2D_64to32(pixel *dst, pixel *src, intptr_t stride)1063
;-----------------------------------------------------------------1064
1065
PIXELSUB_PS_W32_H8_avx2 32, 641066
%endif1067
1068
+%macro PROCESS_SUB_PS_32x8_AVX512 01069
+ pmovzxbw m0, [r2]1070
+ pmovzxbw m1, [r3]1071
+ pmovzxbw m2, [r2 + r4]1072
+ pmovzxbw m3, [r3 + r5]1073
+ pmovzxbw m4, [r2 + 2 * r4]1074
+ pmovzxbw m5, [r3 + 2 * r5]1075
+ pmovzxbw m6, [r2 + r7]1076
+ pmovzxbw m7, [r3 + r8]1077
+1078
+ psubw m0, m11079
+ psubw m2, m31080
+ psubw m4, m51081
+ psubw m6, m71082
+1083
+ movu [r0], m01084
+ movu [r0 + r1], m21085
+ movu [r0 + r1 * 2 ], m41086
+ movu [r0 + r9], m61087
+1088
+ lea r2, [r2 + r4 * 4]1089
+ lea r3, [r3 + r5 * 4]1090
+ lea r0, [r0 + r1 * 4]1091
+1092
+ pmovzxbw m0, [r2]1093
+ pmovzxbw m1, [r3]1094
+ pmovzxbw m2, [r2 + r4]1095
+ pmovzxbw m3, [r3 + r5]1096
+ pmovzxbw m4, [r2 + 2 * r4]1097
+ pmovzxbw m5, [r3 + 2 * r5]1098
+ pmovzxbw m6, [r2 + r7]1099
+ pmovzxbw m7, [r3 + r8]1100
+1101
+ psubw m0, m11102
+ psubw m2, m31103
+ psubw m4, m51104
+ psubw m6, m71105
+1106
+ movu [r0], m01107
+ movu [r0 + r1], m21108
+ movu [r0 + r1 * 2 ], m41109
+ movu [r0 + r9], m61110
+%endmacro1111
+1112
+%macro PROCESS_SUB_PS_32x8_HBD_AVX512 01113
+ movu m0, [r2]1114
+ movu m1, [r3]1115
+ movu m2, [r2 + r4]1116
+ movu m3, [r3 + r5]1117
+ psubw m0, m11118
+ psubw m2, m31119
+1120
+ movu [r0], m01121
+ movu [r0 + r1], m21122
+1123
+ movu m0, [r2 + r4 * 2]1124
+ movu m1, [r3 + r5 * 2]1125
+ movu m2, [r2 + r7]1126
+ movu m3, [r3 + r8]1127
+ psubw m0, m11128
+ psubw m2, m31129
+1130
+ movu [r0 + r1 * 2], m01131
+ movu [r0 + r6], m21132
+1133
+ lea r0, [r0 + r1 * 4]1134
+ lea r2, [r2 + r4 * 4]1135
+ lea r3, [r3 + r5 * 4]1136
+1137
+ movu m0, [r2]1138
+ movu m1, [r3]1139
+ movu m2, [r2 + r4]1140
+ movu m3, [r3 + r5]1141
+ psubw m0, m11142
+ psubw m2, m31143
+1144
+ movu [r0], m01145
+ movu [r0 + r1], m21146
+1147
+ movu m0, [r2 + r4 * 2]1148
+ movu m1, [r3 + r5 * 2]1149
+ movu m2, [r2 + r7]1150
+ movu m3, [r3 + r8]1151
+ psubw m0, m11152
+ psubw m2, m31153
+1154
+ movu [r0 + r1 * 2], m01155
+ movu [r0 + r6], m21156
+%endmacro1157
+1158
+;-----------------------------------------------------------------------------1159
+; void pixel_sub_ps_32x32(int16_t *dest, intptr_t destride, pixel *src0, pixel *src1, intptr_t srcstride0, intptr_t srcstride1);1160
+;-----------------------------------------------------------------------------1161
+%if HIGH_BIT_DEPTH1162
+%if ARCH_X86_641163
+INIT_ZMM avx5121164
+cglobal pixel_sub_ps_32x32, 6, 9, 41165
+ add r1d, r1d1166
+ add r4d, r4d1167
+ add r5d, r5d1168
+ lea r6, [r1 * 3]1169
+ lea r7, [r4 * 3]1170
+ lea r8, [r5 * 3]1171
+ PROCESS_SUB_PS_32x8_HBD_AVX5121172
+ lea r0, [r0 + r1 * 4]1173
+ lea r2, [r2 + r4 * 4]1174
+ lea r3, [r3 + r5 * 4]1175
+ PROCESS_SUB_PS_32x8_HBD_AVX5121176
+ lea r0, [r0 + r1 * 4]1177
+ lea r2, [r2 + r4 * 4]1178
+ lea r3, [r3 + r5 * 4]1179
+ PROCESS_SUB_PS_32x8_HBD_AVX5121180
+ lea r0, [r0 + r1 * 4]1181
+ lea r2, [r2 + r4 * 4]1182
+ lea r3, [r3 + r5 * 4]1183
+ PROCESS_SUB_PS_32x8_HBD_AVX5121184
+ RET1185
+1186
+cglobal pixel_sub_ps_32x64, 6, 9, 41187
+ add r1d, r1d1188
+ add r4d, r4d1189
+ add r5d, r5d1190
+ lea r6, [r1 * 3]1191
+ lea r7, [r4 * 3]1192
+ lea r8, [r5 * 3]1193
+ PROCESS_SUB_PS_32x8_HBD_AVX5121194
+ lea r0, [r0 + r1 * 4]1195
+ lea r2, [r2 + r4 * 4]1196
+ lea r3, [r3 + r5 * 4]1197
+ PROCESS_SUB_PS_32x8_HBD_AVX5121198
+ lea r0, [r0 + r1 * 4]1199
+ lea r2, [r2 + r4 * 4]1200
+ lea r3, [r3 + r5 * 4]1201
+ PROCESS_SUB_PS_32x8_HBD_AVX5121202
+ lea r0, [r0 + r1 * 4]1203
+ lea r2, [r2 + r4 * 4]1204
+ lea r3, [r3 + r5 * 4]1205
+ PROCESS_SUB_PS_32x8_HBD_AVX5121206
+ lea r0, [r0 + r1 * 4]1207
+ lea r2, [r2 + r4 * 4]1208
+ lea r3, [r3 + r5 * 4]1209
+ PROCESS_SUB_PS_32x8_HBD_AVX5121210
+ lea r0, [r0 + r1 * 4]1211
+ lea r2, [r2 + r4 * 4]1212
+ lea r3, [r3 + r5 * 4]1213
+ PROCESS_SUB_PS_32x8_HBD_AVX5121214
+ lea r0, [r0 + r1 * 4]1215
+ lea r2, [r2 + r4 * 4]1216
+ lea r3, [r3 + r5 * 4]1217
+ PROCESS_SUB_PS_32x8_HBD_AVX5121218
+ lea r0, [r0 + r1 * 4]1219
+ lea r2, [r2 + r4 * 4]1220
+ lea r3, [r3 + r5 * 4]1221
+ PROCESS_SUB_PS_32x8_HBD_AVX5121222
+ RET1223
+%endif1224
+%else1225
+%if ARCH_X86_641226
+INIT_ZMM avx5121227
+cglobal pixel_sub_ps_32x32, 6, 10, 81228
+ add r1, r11229
+ lea r7, [r4 * 3]1230
+ lea r8, [r5 * 3]1231
+ lea r9, [r1 * 3]1232
+1233
+ PROCESS_SUB_PS_32x8_AVX5121234
+ lea r2, [r2 + r4 * 4]1235
+ lea r3, [r3 + r5 * 4]1236
+ lea r0, [r0 + r1 * 4]1237
+ PROCESS_SUB_PS_32x8_AVX5121238
+ lea r2, [r2 + r4 * 4]1239
+ lea r3, [r3 + r5 * 4]1240
+ lea r0, [r0 + r1 * 4]1241
+ PROCESS_SUB_PS_32x8_AVX5121242
+ lea r2, [r2 + r4 * 4]1243
+ lea r3, [r3 + r5 * 4]1244
+ lea r0, [r0 + r1 * 4]1245
+ PROCESS_SUB_PS_32x8_AVX5121246
+ RET1247
+1248
+INIT_ZMM avx5121249
+cglobal pixel_sub_ps_32x64, 6, 10, 81250
+ add r1, r11251
+ lea r7, [r4 * 3]1252
+ lea r8, [r5 * 3]1253
+ lea r9, [r1 * 3]1254
+1255
+ PROCESS_SUB_PS_32x8_AVX5121256
+ lea r2, [r2 + r4 * 4]1257
+ lea r3, [r3 + r5 * 4]1258
+ lea r0, [r0 + r1 * 4]1259
+ PROCESS_SUB_PS_32x8_AVX5121260
+ lea r2, [r2 + r4 * 4]1261
+ lea r3, [r3 + r5 * 4]1262
+ lea r0, [r0 + r1 * 4]1263
+ PROCESS_SUB_PS_32x8_AVX5121264
+ lea r2, [r2 + r4 * 4]1265
+ lea r3, [r3 + r5 * 4]1266
+ lea r0, [r0 + r1 * 4]1267
+ PROCESS_SUB_PS_32x8_AVX5121268
+ lea r2, [r2 + r4 * 4]1269
+ lea r3, [r3 + r5 * 4]1270
+ lea r0, [r0 + r1 * 4]1271
+ PROCESS_SUB_PS_32x8_AVX5121272
+ lea r2, [r2 + r4 * 4]1273
+ lea r3, [r3 + r5 * 4]1274
+ lea r0, [r0 + r1 * 4]1275
+ PROCESS_SUB_PS_32x8_AVX5121276
+ lea r2, [r2 + r4 * 4]1277
+ lea r3, [r3 + r5 * 4]1278
+ lea r0, [r0 + r1 * 4]1279
+ PROCESS_SUB_PS_32x8_AVX5121280
+ lea r2, [r2 + r4 * 4]1281
+ lea r3, [r3 + r5 * 4]1282
+ lea r0, [r0 + r1 * 4]1283
+ PROCESS_SUB_PS_32x8_AVX5121284
+ RET1285
+%endif1286
+%endif1287
+1288
;-----------------------------------------------------------------------------1289
; void pixel_sub_ps_64x%2(int16_t *dest, intptr_t destride, pixel *src0, pixel *src1, intptr_t srcstride0, intptr_t srcstride1);1290
;-----------------------------------------------------------------------------1291
1292
jnz .loop1293
RET1294
%endif1295
+1296
+%macro PROCESS_SUB_PS_64x8_AVX512 01297
+ pmovzxbw m0, [r2]1298
+ pmovzxbw m1, [r2 + 32]1299
+ pmovzxbw m2, [r3]1300
+ pmovzxbw m3, [r3 + 32]1301
+ pmovzxbw m4, [r2 + r4]1302
+ pmovzxbw m5, [r2 + r4 + 32]1303
+ pmovzxbw m6, [r3 + r5]1304
+ pmovzxbw m7, [r3 + r5 + 32]1305
+1306
+ psubw m0, m21307
+ psubw m1, m31308
+ psubw m4, m61309
+ psubw m5, m71310
+ movu [r0], m01311
+ movu [r0 + 64], m11312
+ movu [r0 + 2 * r1], m41313
+ movu [r0 + 2 * r1 + 64], m51314
+1315
+ lea r0, [r0 + 4 * r1]1316
+ lea r2, [r2 + 2 * r4]1317
+ lea r3, [r3 + 2 * r5]1318
+1319
+ pmovzxbw m0, [r2]1320
+ pmovzxbw m1, [r2 + 32]1321
+ pmovzxbw m2, [r3]1322
+ pmovzxbw m3, [r3 + 32]1323
+ pmovzxbw m4, [r2 + r4]1324
+ pmovzxbw m5, [r2 + r4 + 32]1325
+ pmovzxbw m6, [r3 + r5]1326
+ pmovzxbw m7, [r3 + r5 + 32]1327
+1328
+ psubw m0, m21329
+ psubw m1, m31330
+ psubw m4, m61331
+ psubw m5, m71332
+ movu [r0], m01333
+ movu [r0 + 64], m11334
+ movu [r0 + 2 * r1], m41335
+ movu [r0 + 2 * r1 + 64], m51336
+1337
+ lea r0, [r0 + 4 * r1]1338
+ lea r2, [r2 + 2 * r4]1339
+ lea r3, [r3 + 2 * r5]1340
+1341
+ pmovzxbw m0, [r2]1342
+ pmovzxbw m1, [r2 + 32]1343
+ pmovzxbw m2, [r3]1344
+ pmovzxbw m3, [r3 + 32]1345
+ pmovzxbw m4, [r2 + r4]1346
+ pmovzxbw m5, [r2 + r4 + 32]1347
+ pmovzxbw m6, [r3 + r5]1348
+ pmovzxbw m7, [r3 + r5 + 32]1349
+1350
+ psubw m0, m21351
+ psubw m1, m31352
+ psubw m4, m61353
+ psubw m5, m71354
+ movu [r0], m01355
+ movu [r0 + 64], m11356
+ movu [r0 + 2 * r1], m41357
+ movu [r0 + 2 * r1 + 64], m51358
+1359
+ lea r0, [r0 + 4 * r1]1360
+ lea r2, [r2 + 2 * r4]1361
+ lea r3, [r3 + 2 * r5]1362
+1363
+ pmovzxbw m0, [r2]1364
+ pmovzxbw m1, [r2 + 32]1365
+ pmovzxbw m2, [r3]1366
+ pmovzxbw m3, [r3 + 32]1367
+ pmovzxbw m4, [r2 + r4]1368
+ pmovzxbw m5, [r2 + r4 + 32]1369
+ pmovzxbw m6, [r3 + r5]1370
+ pmovzxbw m7, [r3 + r5 + 32]1371
+1372
+ psubw m0, m21373
+ psubw m1, m31374
+ psubw m4, m61375
+ psubw m5, m71376
+ movu [r0], m01377
+ movu [r0 + 64], m11378
+ movu [r0 + 2 * r1], m41379
+ movu [r0 + 2 * r1 + 64], m51380
+%endmacro1381
+1382
+%macro PROCESS_SUB_PS_64x8_HBD_AVX512 01383
+ movu m0, [r2]1384
+ movu m1, [r2 + 64]1385
+ movu m4, [r3]1386
+ movu m5, [r3 + 64]1387
+ psubw m0, m41388
+ psubw m1, m51389
+ movu m2, [r2 + r4]1390
+ movu m3, [r2 + r4 + 64]1391
+ movu m6, [r3 + r5]1392
+ movu m7, [r3 + r5 + 64]1393
+ psubw m2, m61394
+ psubw m3, m71395
+1396
+ movu [r0], m01397
+ movu [r0 + 64], m11398
+ movu [r0 + r1], m21399
+ movu [r0 + r1 + 64], m31400
+1401
+ movu m0, [r2 + r4 * 2]1402
+ movu m1, [r2 + r4 * 2 + 64]1403
+ movu m4, [r3 + r5 * 2]1404
+ movu m5, [r3 + r5 * 2 + 64]1405
+ psubw m0, m41406
+ psubw m1, m51407
+ movu m2, [r2 + r7]1408
+ movu m3, [r2 + r7 + 64]1409
+ movu m6, [r3 + r8]1410
+ movu m7, [r3 + r8 + 64]1411
+ psubw m2, m61412
+ psubw m3, m71413
+1414
+ movu [r0 + r1 * 2], m01415
+ movu [r0 + r1 * 2 + 64], m11416
+ movu [r0 + r6], m21417
+ movu [r0 + r6 + 64], m31418
+1419
+ lea r0, [r0 + r1 * 4]1420
+ lea r2, [r2 + r4 * 4]1421
+ lea r3, [r3 + r5 * 4]1422
+1423
+ movu m0, [r2]1424
+ movu m1, [r2 + 64]1425
+ movu m4, [r3]1426
+ movu m5, [r3 + 64]1427
+ psubw m0, m41428
+ psubw m1, m51429
+ movu m2, [r2 + r4]1430
+ movu m3, [r2 + r4 + 64]1431
+ movu m6, [r3 + r5]1432
+ movu m7, [r3 + r5 + 64]1433
+ psubw m2, m61434
+ psubw m3, m71435
+1436
+ movu [r0], m01437
+ movu [r0 + 64], m11438
+ movu [r0 + r1], m21439
+ movu [r0 + r1 + 64], m31440
+1441
+ movu m0, [r2 + r4 * 2]1442
+ movu m1, [r2 + r4 * 2 + 64]1443
+ movu m4, [r3 + r5 * 2]1444
+ movu m5, [r3 + r5 * 2 + 64]1445
+ psubw m0, m41446
+ psubw m1, m51447
+ movu m2, [r2 + r7]1448
+ movu m3, [r2 + r7 + 64]1449
+ movu m6, [r3 + r8]1450
+ movu m7, [r3 + r8 + 64]1451
+ psubw m2, m61452
+ psubw m3, m71453
+1454
+ movu [r0 + r1 * 2], m01455
+ movu [r0 + r1 * 2 + 64], m11456
+ movu [r0 + r6], m21457
+ movu [r0 + r6 + 64], m31458
+%endmacro1459
+;-----------------------------------------------------------------------------1460
+; void pixel_sub_ps_64x64(int16_t *dest, intptr_t destride, pixel *src0, pixel *src1, intptr_t srcstride0, intptr_t srcstride1);1461
+;-----------------------------------------------------------------------------1462
+%if HIGH_BIT_DEPTH1463
+%if ARCH_X86_641464
+INIT_ZMM avx5121465
+cglobal pixel_sub_ps_64x64, 6, 9, 81466
+ add r1d, r1d1467
+ add r4d, r4d1468
+ add r5d, r5d1469
+ lea r6, [r1 * 3]1470
+ lea r7, [r4 * 3]1471
+ lea r8, [r5 * 3]1472
+1473
+ PROCESS_SUB_PS_64x8_HBD_AVX5121474
+ lea r0, [r0 + r1 * 4]1475
+ lea r2, [r2 + r4 * 4]1476
+ lea r3, [r3 + r5 * 4]1477
+ PROCESS_SUB_PS_64x8_HBD_AVX5121478
+ lea r0, [r0 + r1 * 4]1479
+ lea r2, [r2 + r4 * 4]1480
+ lea r3, [r3 + r5 * 4]1481
+ PROCESS_SUB_PS_64x8_HBD_AVX5121482
+ lea r0, [r0 + r1 * 4]1483
+ lea r2, [r2 + r4 * 4]1484
+ lea r3, [r3 + r5 * 4]1485
+ PROCESS_SUB_PS_64x8_HBD_AVX5121486
+ lea r0, [r0 + r1 * 4]1487
+ lea r2, [r2 + r4 * 4]1488
+ lea r3, [r3 + r5 * 4]1489
+ PROCESS_SUB_PS_64x8_HBD_AVX5121490
+ lea r0, [r0 + r1 * 4]1491
+ lea r2, [r2 + r4 * 4]1492
+ lea r3, [r3 + r5 * 4]1493
+ PROCESS_SUB_PS_64x8_HBD_AVX5121494
+ lea r0, [r0 + r1 * 4]1495
+ lea r2, [r2 + r4 * 4]1496
+ lea r3, [r3 + r5 * 4]1497
+ PROCESS_SUB_PS_64x8_HBD_AVX5121498
+ lea r0, [r0 + r1 * 4]1499
+ lea r2, [r2 + r4 * 4]1500
+ lea r3, [r3 + r5 * 4]1501
+ PROCESS_SUB_PS_64x8_HBD_AVX5121502
+ RET1503
+%endif1504
+%else1505
+%if ARCH_X86_641506
+INIT_ZMM avx5121507
+cglobal pixel_sub_ps_64x64, 6, 7, 81508
+ PROCESS_SUB_PS_64x8_AVX5121509
+ lea r0, [r0 + 4 * r1]1510
+ lea r2, [r2 + 2 * r4]1511
+ lea r3, [r3 + 2 * r5]1512
+ PROCESS_SUB_PS_64x8_AVX5121513
+ lea r0, [r0 + 4 * r1]1514
+ lea r2, [r2 + 2 * r4]1515
+ lea r3, [r3 + 2 * r5]1516
+ PROCESS_SUB_PS_64x8_AVX5121517
+ lea r0, [r0 + 4 * r1]1518
+ lea r2, [r2 + 2 * r4]1519
+ lea r3, [r3 + 2 * r5]1520
+ PROCESS_SUB_PS_64x8_AVX5121521
+ lea r0, [r0 + 4 * r1]1522
+ lea r2, [r2 + 2 * r4]1523
+ lea r3, [r3 + 2 * r5]1524
+ PROCESS_SUB_PS_64x8_AVX5121525
+ lea r0, [r0 + 4 * r1]1526
+ lea r2, [r2 + 2 * r4]1527
+ lea r3, [r3 + 2 * r5]1528
+ PROCESS_SUB_PS_64x8_AVX5121529
+ lea r0, [r0 + 4 * r1]1530
+ lea r2, [r2 + 2 * r4]1531
+ lea r3, [r3 + 2 * r5]1532
+ PROCESS_SUB_PS_64x8_AVX5121533
+ lea r0, [r0 + 4 * r1]1534
+ lea r2, [r2 + 2 * r4]1535
+ lea r3, [r3 + 2 * r5]1536
+ PROCESS_SUB_PS_64x8_AVX5121537
+ RET1538
+%endif1539
+%endif1540
;=============================================================================1541
; variance1542
;=============================================================================1543
1544
%if HIGH_BIT_DEPTH == 01545
%if %11546
mova m7, [pw_00ff]1547
-%elif mmsize < 321548
+%elif mmsize == 161549
pxor m7, m7 ; zero1550
%endif1551
%endif ; !HIGH_BIT_DEPTH1552
1553
RET1554
%endif ; !HIGH_BIT_DEPTH1555
1556
+%macro PROCESS_VAR_32x8_AVX512 01557
+ pmovzxbw m0, [r0]1558
+ pmovzxbw m1, [r0 + r1]1559
+ pmovzxbw m2, [r0 + 2 * r1]1560
+ pmovzxbw m3, [r0 + r2]1561
+1562
+ paddw m4, m01563
+ paddw m4, m11564
+ paddw m4, m21565
+ paddw m4, m31566
+ pmaddwd m0, m01567
+ pmaddwd m1, m11568
+ pmaddwd m2, m21569
+ pmaddwd m3, m31570
+ paddd m5, m01571
+ paddd m5, m11572
+ paddd m5, m21573
+ paddd m5, m31574
+1575
+ lea r0, [r0 + r1 * 4]1576
+1577
+ pmovzxbw m0, [r0]1578
+ pmovzxbw m1, [r0 + r1]1579
+ pmovzxbw m2, [r0 + 2 * r1]1580
+ pmovzxbw m3, [r0 + r2]1581
+1582
+ paddw m4, m01583
+ paddw m4, m11584
+ paddw m4, m21585
+ paddw m4, m31586
+ pmaddwd m0, m01587
+ pmaddwd m1, m11588
+ pmaddwd m2, m21589
+ pmaddwd m3, m31590
+ paddd m5, m01591
+ paddd m5, m11592
+ paddd m5, m21593
+ paddd m5, m31594
+%endmacro1595
+1596
+%macro PROCESS_VAR_AVX512_END 01597
+ vextracti32x8 ym0, m4, 11598
+ vextracti32x8 ym1, m5, 11599
+ paddw ym4, ym01600
+ paddd ym5, ym11601
+ vextracti32x4 xm0, m4, 11602
+ vextracti32x4 xm1, m5, 11603
+ paddw xm4, xm01604
+ paddd xm5, xm11605
+ HADDW xm4, xm21606
+ HADDD xm5, xm11607
+%if ARCH_X86_641608
+ punpckldq xm4, xm51609
+ movq rax, xm41610
+%else1611
+ movd eax, xm41612
+ movd edx, xm51613
+%endif1614
+%endmacro1615
+%if ARCH_X86_64 == 1 && HIGH_BIT_DEPTH == 01616
+;-----------------------------------------------------------------------------1617
+; int pixel_var_wxh( uint8_t *, intptr_t )1618
+;-----------------------------------------------------------------------------1619
+INIT_ZMM avx5121620
+cglobal pixel_var_32x32, 2,4,61621
+ pxor m4, m4 ; sum1622
+ pxor m5, m5 ; sum squared1623
+ lea r2, [3 * r1]1624
+1625
+ PROCESS_VAR_32x8_AVX5121626
+ lea r0, [r0 + r1 * 4]1627
+ PROCESS_VAR_32x8_AVX5121628
+ lea r0, [r0 + r1 * 4]1629
+ PROCESS_VAR_32x8_AVX5121630
+ lea r0, [r0 + r1 * 4]1631
+ PROCESS_VAR_32x8_AVX5121632
+ PROCESS_VAR_AVX512_END1633
+ RET1634
+1635
+INIT_ZMM avx5121636
+cglobal pixel_var_64x64, 2,4,71637
+ pxor m5, m5 ; sum1638
+ pxor m6, m6 ; sum squared1639
+ mov r2d, 321640
+1641
+.loop:1642
+ pmovzxbw m0, [r0]1643
+ pmovzxbw m3, [r0 + mmsize/2]1644
+ pmovzxbw m1, [r0 + r1]1645
+ pmovzxbw m4, [r0 + r1 + mmsize/2]1646
+1647
+ lea r0, [r0 + 2 * r1]1648
+1649
+ paddw m5, m01650
+ paddw m5, m31651
+ paddw m5, m11652
+ paddw m5, m41653
+ pmaddwd m0, m01654
+ pmaddwd m3, m31655
+ pmaddwd m1, m11656
+ pmaddwd m4, m41657
+ paddd m6, m01658
+ paddd m6, m31659
+ paddd m6, m11660
+ paddd m6, m41661
+1662
+ dec r2d1663
+ jg .loop1664
+1665
+ pxor m1, m11666
+ punpcklwd m0, m5, m11667
+ punpckhwd m5, m11668
+ paddd m5, m01669
+ vextracti32x8 ym2, m5, 11670
+ vextracti32x8 ym1, m6, 11671
+ paddd ym5, ym21672
+ paddd ym6, ym11673
+ vextracti32x4 xm2, m5, 11674
+ vextracti32x4 xm1, m6, 11675
+ paddd xm5, xm21676
+ paddd xm6, xm11677
+ HADDD xm5, xm21678
+ HADDD xm6, xm11679
+ punpckldq xm5, xm61680
+ movq rax, xm51681
+ RET1682
+%endif1683
+%macro VAR_AVX512_CORE 1 ; accum1684
+%if %11685
+ paddw m0, m21686
+ pmaddwd m2, m21687
+ paddw m0, m31688
+ pmaddwd m3, m31689
+ paddd m1, m21690
+ paddd m1, m31691
+%else1692
+ paddw m0, m2, m31693
+ pmaddwd m2, m21694
+ pmaddwd m3, m31695
+ paddd m1, m2, m31696
+%endif1697
+%endmacro1698
+1699
+%macro VAR_AVX512_CORE_16x16 1 ; accum1700
+%if HIGH_BIT_DEPTH1701
+ mova ym2, [r0]1702
+ vinserti64x4 m2, [r0+r1], 11703
+ mova ym3, [r0+2*r1]1704
+ vinserti64x4 m3, [r0+r3], 11705
+%else1706
+ vbroadcasti64x2 ym2, [r0]1707
+ vbroadcasti64x2 m2 {k1}, [r0+r1]1708
+ vbroadcasti64x2 ym3, [r0+2*r1]1709
+ vbroadcasti64x2 m3 {k1}, [r0+r3]1710
+ pshufb m2, m41711
+ pshufb m3, m41712
+%endif1713
+ VAR_AVX512_CORE %11714
+%endmacro1715
+1716
+%macro VAR_AVX512_CORE_8x8 1 ; accum1717
+%if HIGH_BIT_DEPTH1718
+ mova xm2, [r0]1719
+ mova xm3, [r0+r1]1720
+%else1721
+ movq xm2, [r0]1722
+ movq xm3, [r0+r1]1723
+%endif1724
+ vinserti128 ym2, [r0+2*r1], 11725
+ vinserti128 ym3, [r0+r2], 11726
+ lea r0, [r0+4*r1]1727
+ vinserti32x4 m2, [r0], 21728
+ vinserti32x4 m3, [r0+r1], 21729
+ vinserti32x4 m2, [r0+2*r1], 31730
+ vinserti32x4 m3, [r0+r2], 31731
+%if HIGH_BIT_DEPTH == 01732
+ punpcklbw m2, m41733
+ punpcklbw m3, m41734
+%endif1735
+ VAR_AVX512_CORE %11736
+%endmacro1737
+1738
+INIT_ZMM avx5121739
+cglobal pixel_var_16x16, 2,41740
+ FIX_STRIDES r11741
+ mov r2d, 0xf01742
+ lea r3, [3*r1]1743
+%if HIGH_BIT_DEPTH == 01744
+ vbroadcasti64x4 m4, [var_shuf_avx512]1745
+ kmovb k1, r2d1746
+%endif1747
+ VAR_AVX512_CORE_16x16 01748
+.loop:1749
+ lea r0, [r0+4*r1]1750
+ VAR_AVX512_CORE_16x16 11751
+ sub r2d, 0x501752
+ jg .loop1753
+%if ARCH_X86_64 == 01754
+ pop r3d1755
+ %assign regs_used 31756
+%endif1757
+var_avx512_end:1758
+ vbroadcasti32x4 m2, [pw_1]1759
+ pmaddwd m0, m21760
+ SBUTTERFLY dq, 0, 1, 21761
+ paddd m0, m11762
+ vextracti32x8 ym1, m0, 11763
+ paddd ym0, ym11764
+ vextracti128 xm1, ym0, 11765
+ paddd xmm0, xm0, xm11766
+ punpckhqdq xmm1, xmm0, xmm01767
+ paddd xmm0, xmm11768
+%if ARCH_X86_641769
+ movq rax, xmm01770
+%else1771
+ movd eax, xmm01772
+ pextrd edx, xmm0, 11773
+ %endif1774
+ RET1775
+ 1776
+%if HIGH_BIT_DEPTH == 0 ; 8x8 doesn't benefit from AVX-512 in high bit-depth1777
+cglobal pixel_var_8x8, 2,31778
+ lea r2, [3*r1]1779
+ pxor xm4, xm41780
+ VAR_AVX512_CORE_8x8 01781
+ jmp var_avx512_end1782
+%endif1783
+1784
+cglobal pixel_var_8x16, 2,31785
+ FIX_STRIDES r11786
+ lea r2, [3*r1]1787
+%if HIGH_BIT_DEPTH == 01788
+ pxor xm4, xm41789
+%endif1790
+ VAR_AVX512_CORE_8x8 01791
+ lea r0, [r0+4*r1]1792
+ VAR_AVX512_CORE_8x8 11793
+ jmp var_avx512_end1794
+1795
%macro VAR2_END 31796
HADDW %2, xm11797
movd r1d, %21798
x265_2.7.tar.gz/source/common/x86/pixel.h -> x265_2.9.tar.gz/source/common/x86/pixel.h
Changed
37
1
2
void PFX(downShift_16_avx2)(const uint16_t* src, intptr_t srcStride, pixel* dst, intptr_t dstStride, int width, int height, int shift, uint16_t mask);3
void PFX(upShift_16_sse2)(const uint16_t* src, intptr_t srcStride, pixel* dst, intptr_t dstStride, int width, int height, int shift, uint16_t mask);4
void PFX(upShift_16_avx2)(const uint16_t* src, intptr_t srcStride, pixel* dst, intptr_t dstStride, int width, int height, int shift, uint16_t mask);5
+void PFX(upShift_16_avx512)(const uint16_t* src, intptr_t srcStride, pixel* dst, intptr_t dstStride, int width, int height, int shift, uint16_t mask);6
void PFX(upShift_8_sse4)(const uint8_t* src, intptr_t srcStride, pixel* dst, intptr_t dstStride, int width, int height, int shift);7
void PFX(upShift_8_avx2)(const uint8_t* src, intptr_t srcStride, pixel* dst, intptr_t dstStride, int width, int height, int shift);8
pixel PFX(planeClipAndMax_avx2)(pixel *src, intptr_t stride, int width, int height, uint64_t *outsum, const pixel minPix, const pixel maxPix);9
10
FUNCDEF_PU(void, pixel_sad_x3, cpu, const pixel*, const pixel*, const pixel*, const pixel*, intptr_t, int32_t*); \11
FUNCDEF_PU(void, pixel_sad_x4, cpu, const pixel*, const pixel*, const pixel*, const pixel*, const pixel*, intptr_t, int32_t*); \12
FUNCDEF_PU(void, pixel_avg, cpu, pixel* dst, intptr_t dstride, const pixel* src0, intptr_t sstride0, const pixel* src1, intptr_t sstride1, int); \13
+ FUNCDEF_PU(void, pixel_avg_aligned, cpu, pixel* dst, intptr_t dstride, const pixel* src0, intptr_t sstride0, const pixel* src1, intptr_t sstride1, int); \14
FUNCDEF_PU(void, pixel_add_ps, cpu, pixel* a, intptr_t dstride, const pixel* b0, const int16_t* b1, intptr_t sstride0, intptr_t sstride1); \15
+ FUNCDEF_PU(void, pixel_add_ps_aligned, cpu, pixel* a, intptr_t dstride, const pixel* b0, const int16_t* b1, intptr_t sstride0, intptr_t sstride1); \16
FUNCDEF_PU(void, pixel_sub_ps, cpu, int16_t* a, intptr_t dstride, const pixel* b0, const pixel* b1, intptr_t sstride0, intptr_t sstride1); \17
FUNCDEF_CHROMA_PU(int, pixel_satd, cpu, const pixel*, intptr_t, const pixel*, intptr_t); \18
FUNCDEF_CHROMA_PU(int, pixel_sad, cpu, const pixel*, intptr_t, const pixel*, intptr_t); \19
FUNCDEF_CHROMA_PU(sse_t, pixel_ssd_ss, cpu, const int16_t*, intptr_t, const int16_t*, intptr_t); \20
FUNCDEF_CHROMA_PU(void, addAvg, cpu, const int16_t*, const int16_t*, pixel*, intptr_t, intptr_t, intptr_t); \21
+ FUNCDEF_CHROMA_PU(void, addAvg_aligned, cpu, const int16_t*, const int16_t*, pixel*, intptr_t, intptr_t, intptr_t); \22
FUNCDEF_CHROMA_PU(sse_t, pixel_ssd_s, cpu, const int16_t*, intptr_t); \23
+ FUNCDEF_CHROMA_PU(sse_t, pixel_ssd_s_aligned, cpu, const int16_t*, intptr_t); \24
FUNCDEF_TU_S(sse_t, pixel_ssd_s, cpu, const int16_t*, intptr_t); \25
+ FUNCDEF_TU_S(sse_t, pixel_ssd_s_aligned, cpu, const int16_t*, intptr_t); \26
FUNCDEF_TU(uint64_t, pixel_var, cpu, const pixel*, intptr_t); \27
FUNCDEF_TU(int, psyCost_pp, cpu, const pixel* source, intptr_t sstride, const pixel* recon, intptr_t rstride); \28
FUNCDEF_TU(int, psyCost_ss, cpu, const int16_t* source, intptr_t sstride, const int16_t* recon, intptr_t rstride)29
30
DECL_PIXELS(avx);31
DECL_PIXELS(xop);32
DECL_PIXELS(avx2);33
+DECL_PIXELS(avx512);34
35
#undef DECL_PIXELS36
37
x265_2.7.tar.gz/source/common/x86/pixeladd8.asm -> x265_2.9.tar.gz/source/common/x86/pixeladd8.asm
Changed
530
1
2
3
%include "x86inc.asm"4
%include "x86util.asm"5
+SECTION_RODATA 646
7
-SECTION_RODATA 328
-9
+ALIGN 6410
+const store_shuf1_avx512, dq 0, 2, 4, 6, 1, 3, 5, 711
SECTION .text12
-13
cextern pw_pixel_max14
15
;-----------------------------------------------------------------------------16
17
PIXEL_ADD_PS_W32_H4_avx2 3218
PIXEL_ADD_PS_W32_H4_avx2 6419
20
-21
;-----------------------------------------------------------------------------22
; void pixel_add_ps_64x%2(pixel *dest, intptr_t destride, pixel *src0, int16_t *scr1, intptr_t srcStride0, intptr_t srcStride1)23
;-----------------------------------------------------------------------------24
25
RET26
27
%endif28
+29
+;-----------------------------------------------------------------------------30
+; pixel_add_ps avx512 code start31
+;-----------------------------------------------------------------------------32
+%macro PROCESS_ADD_PS_64x4_AVX512 033
+ pmovzxbw m0, [r2]34
+ pmovzxbw m1, [r2 + mmsize/2]35
+ movu m2, [r3]36
+ movu m3, [r3 + mmsize]37
+ paddw m0, m238
+ paddw m1, m339
+ packuswb m0, m140
+ vpermq m0, m4, m041
+ movu [r0], m042
+ pmovzxbw m0, [r2 + r4]43
+ pmovzxbw m1, [r2 + r4 + mmsize/2]44
+ movu m2, [r3 + r5]45
+ movu m3, [r3 + r5 + mmsize]46
+ paddw m0, m247
+ paddw m1, m348
+ packuswb m0, m149
+ vpermq m0, m4, m050
+ movu [r0 + r1], m051
+ pmovzxbw m0, [r2 + 2 * r4]52
+ pmovzxbw m1, [r2 + 2 * r4 + mmsize/2]53
+ movu m2, [r3 + 2 * r5]54
+ movu m3, [r3 + 2 * r5 + mmsize]55
+ paddw m0, m256
+ paddw m1, m357
+ packuswb m0, m158
+ vpermq m0, m4, m059
+ movu [r0 + 2 * r1], m060
+61
+ pmovzxbw m0, [r2 + r7]62
+ pmovzxbw m1, [r2 + r7 + mmsize/2]63
+ movu m2, [r3 + r8]64
+ movu m3, [r3 + r8 + mmsize]65
+ paddw m0, m266
+ paddw m1, m367
+ packuswb m0, m168
+ vpermq m0, m4, m069
+ movu [r0 + r6], m070
+%endmacro71
+72
+%macro PROCESS_ADD_PS_64x4_HBD_AVX512 073
+ movu m0, [r2]74
+ movu m1, [r2 + mmsize]75
+ movu m2, [r3]76
+ movu m3, [r3 + mmsize]77
+ paddw m0, m278
+ paddw m1, m379
+80
+ CLIPW2 m0, m1, m4, m581
+ movu [r0], m082
+ movu [r0 + mmsize], m183
+84
+ movu m0, [r2 + r4]85
+ movu m1, [r2 + r4 + mmsize]86
+ movu m2, [r3 + r5]87
+ movu m3, [r3 + r5 + mmsize]88
+ paddw m0, m289
+ paddw m1, m390
+91
+ CLIPW2 m0, m1, m4, m592
+ movu [r0 + r1], m093
+ movu [r0 + r1 + mmsize], m194
+95
+ movu m0, [r2 + r4 * 2]96
+ movu m1, [r2 + r4 * 2 + mmsize]97
+ movu m2, [r3 + r5 * 2]98
+ movu m3, [r3 + r5 * 2 + mmsize]99
+ paddw m0, m2100
+ paddw m1, m3101
+102
+ CLIPW2 m0, m1, m4, m5103
+ movu [r0 + r1 * 2], m0104
+ movu [r0 + r1 * 2 + mmsize], m1105
+106
+ movu m0, [r2 + r6]107
+ movu m1, [r2 + r6 + mmsize]108
+ movu m2, [r3 + r7]109
+ movu m3, [r3 + r7 + mmsize]110
+ paddw m0, m2111
+ paddw m1, m3112
+113
+ CLIPW2 m0, m1, m4, m5114
+ movu [r0 + r8], m0115
+ movu [r0 + r8 + mmsize], m1116
+%endmacro117
+118
+%macro PROCESS_ADD_PS_64x4_ALIGNED_AVX512 0119
+ pmovzxbw m0, [r2]120
+ pmovzxbw m1, [r2 + mmsize/2]121
+ mova m2, [r3]122
+ mova m3, [r3 + mmsize]123
+ paddw m0, m2124
+ paddw m1, m3125
+ packuswb m0, m1126
+ vpermq m0, m4, m0127
+ mova [r0], m0128
+ pmovzxbw m0, [r2 + r4]129
+ pmovzxbw m1, [r2 + r4 + mmsize/2]130
+ mova m2, [r3 + r5]131
+ mova m3, [r3 + r5 + mmsize]132
+ paddw m0, m2133
+ paddw m1, m3134
+ packuswb m0, m1135
+ vpermq m0, m4, m0136
+ mova [r0 + r1], m0137
+ pmovzxbw m0, [r2 + 2 * r4]138
+ pmovzxbw m1, [r2 + 2 * r4 + mmsize/2]139
+ mova m2, [r3 + 2 * r5]140
+ mova m3, [r3 + 2 * r5 + mmsize]141
+ paddw m0, m2142
+ paddw m1, m3143
+ packuswb m0, m1144
+ vpermq m0, m4, m0145
+ mova [r0 + 2 * r1], m0146
+147
+ pmovzxbw m0, [r2 + r7]148
+ pmovzxbw m1, [r2 + r7 + mmsize/2]149
+ mova m2, [r3 + r8]150
+ mova m3, [r3 + r8 + mmsize]151
+ paddw m0, m2152
+ paddw m1, m3153
+ packuswb m0, m1154
+ vpermq m0, m4, m0155
+ mova [r0 + r6], m0156
+%endmacro157
+158
+%macro PROCESS_ADD_PS_64x4_HBD_ALIGNED_AVX512 0159
+ mova m0, [r2]160
+ mova m1, [r2 + mmsize]161
+ mova m2, [r3]162
+ mova m3, [r3 + mmsize]163
+ paddw m0, m2164
+ paddw m1, m3165
+166
+ CLIPW2 m0, m1, m4, m5167
+ mova [r0], m0168
+ mova [r0 + mmsize], m1169
+170
+ mova m0, [r2 + r4]171
+ mova m1, [r2 + r4 + mmsize]172
+ mova m2, [r3 + r5]173
+ mova m3, [r3 + r5 + mmsize]174
+ paddw m0, m2175
+ paddw m1, m3176
+177
+ CLIPW2 m0, m1, m4, m5178
+ mova [r0 + r1], m0179
+ mova [r0 + r1 + mmsize], m1180
+181
+ mova m0, [r2 + r4 * 2]182
+ mova m1, [r2 + r4 * 2 + mmsize]183
+ mova m2, [r3 + r5 * 2]184
+ mova m3, [r3 + r5 * 2 + mmsize]185
+ paddw m0, m2186
+ paddw m1, m3187
+188
+ CLIPW2 m0, m1, m4, m5189
+ mova [r0 + r1 * 2], m0190
+ mova [r0 + r1 * 2 + mmsize], m1191
+192
+ mova m0, [r2 + r6]193
+ mova m1, [r2 + r6 + mmsize]194
+ mova m2, [r3 + r7]195
+ mova m3, [r3 + r7 + mmsize]196
+ paddw m0, m2197
+ paddw m1, m3198
+199
+ CLIPW2 m0, m1, m4, m5200
+ mova [r0 + r8], m0201
+ mova [r0 + r8 + mmsize], m1202
+%endmacro203
+204
+;-----------------------------------------------------------------------------205
+; void pixel_add_ps_64x64(pixel *dest, intptr_t destride, pixel *src0, int16_t *scr1, intptr_t srcStride0, intptr_t srcStride1)206
+;-----------------------------------------------------------------------------207
+%if HIGH_BIT_DEPTH208
+%if ARCH_X86_64209
+INIT_ZMM avx512210
+cglobal pixel_add_ps_64x64, 6, 9, 6211
+ vbroadcasti32x8 m5, [pw_pixel_max]212
+ pxor m4, m4213
+ add r4d, r4d214
+ add r5d, r5d215
+ add r1d, r1d216
+ lea r6, [r4 * 3]217
+ lea r7, [r5 * 3]218
+ lea r8, [r1 * 3]219
+%rep 15220
+ PROCESS_ADD_PS_64x4_HBD_AVX512221
+ lea r2, [r2 + r4 * 4]222
+ lea r3, [r3 + r5 * 4]223
+ lea r0, [r0 + r1 * 4]224
+%endrep225
+ PROCESS_ADD_PS_64x4_HBD_AVX512226
+ RET227
+228
+INIT_ZMM avx512229
+cglobal pixel_add_ps_aligned_64x64, 6, 9, 6230
+ vbroadcasti32x8 m5, [pw_pixel_max]231
+ pxor m4, m4232
+ add r4d, r4d233
+ add r5d, r5d234
+ add r1d, r1d235
+ lea r6, [r4 * 3]236
+ lea r7, [r5 * 3]237
+ lea r8, [r1 * 3]238
+%rep 15239
+ PROCESS_ADD_PS_64x4_HBD_ALIGNED_AVX512240
+ lea r2, [r2 + r4 * 4]241
+ lea r3, [r3 + r5 * 4]242
+ lea r0, [r0 + r1 * 4]243
+%endrep244
+ PROCESS_ADD_PS_64x4_HBD_ALIGNED_AVX512245
+ RET246
+%endif247
+%else248
+%if ARCH_X86_64249
+INIT_ZMM avx512250
+cglobal pixel_add_ps_64x64, 6, 9, 4251
+ add r5, r5252
+ lea r6, [3 * r1]253
+ lea r7, [3 * r4]254
+ lea r8, [3 * r5]255
+ mova m4, [store_shuf1_avx512]256
+%rep 15257
+ PROCESS_ADD_PS_64x4_AVX512258
+ lea r2, [r2 + r4 * 4]259
+ lea r3, [r3 + r5 * 4]260
+ lea r0, [r0 + r1 * 4]261
+%endrep262
+ PROCESS_ADD_PS_64x4_AVX512263
+ RET264
+265
+INIT_ZMM avx512266
+cglobal pixel_add_ps_aligned_64x64, 6, 9, 4267
+ add r5, r5268
+ lea r6, [3 * r1]269
+ lea r7, [3 * r4]270
+ lea r8, [3 * r5]271
+ mova m4, [store_shuf1_avx512]272
+%rep 15273
+ PROCESS_ADD_PS_64x4_ALIGNED_AVX512274
+ lea r2, [r2 + r4 * 4]275
+ lea r3, [r3 + r5 * 4]276
+ lea r0, [r0 + r1 * 4]277
+%endrep278
+ PROCESS_ADD_PS_64x4_ALIGNED_AVX512279
+ RET280
+%endif281
+%endif282
+283
+%macro PROCESS_ADD_PS_32x4_AVX512 0284
+ pmovzxbw m0, [r2]285
+ movu m1, [r3]286
+ pmovzxbw m2, [r2 + r4]287
+ movu m3, [r3 + r5]288
+ paddw m0, m1289
+ paddw m2, m3290
+ packuswb m0, m2291
+ vpermq m0, m4, m0292
+ movu [r0], ym0293
+ vextracti32x8 [r0 + r1], m0, 1294
+ pmovzxbw m0, [r2 + r4 * 2]295
+ movu m1, [r3 + r5 * 2]296
+ pmovzxbw m2, [r2 + r6]297
+ movu m3, [r3 + r7]298
+ paddw m0, m1299
+ paddw m2, m3300
+ packuswb m0, m2301
+ vpermq m0, m4, m0302
+ movu [r0 + r1 * 2], ym0303
+ vextracti32x8 [r0 + r8], m0, 1304
+%endmacro305
+306
+%macro PROCESS_ADD_PS_32x4_HBD_AVX512 0307
+ movu m0, [r2]308
+ movu m1, [r2 + r4]309
+ movu m2, [r3]310
+ movu m3, [r3 + r5]311
+ paddw m0, m2312
+ paddw m1, m3313
+314
+ CLIPW2 m0, m1, m4, m5315
+ movu [r0], m0316
+ movu [r0 + r1], m1317
+318
+ movu m0, [r2 + r4 * 2]319
+ movu m1, [r2 + r6]320
+ movu m2, [r3 + r5 * 2]321
+ movu m3, [r3 + r7]322
+ paddw m0, m2323
+ paddw m1, m3324
+325
+ CLIPW2 m0, m1, m4, m5326
+ movu [r0 + r1 * 2], m0327
+ movu [r0 + r8], m1328
+%endmacro329
+330
+%macro PROCESS_ADD_PS_32x4_ALIGNED_AVX512 0331
+ pmovzxbw m0, [r2]332
+ mova m1, [r3]333
+ pmovzxbw m2, [r2 + r4]334
+ mova m3, [r3 + r5]335
+ paddw m0, m1336
+ paddw m2, m3337
+ packuswb m0, m2338
+ vpermq m0, m4, m0339
+ mova [r0], ym0340
+ vextracti32x8 [r0 + r1], m0, 1341
+ pmovzxbw m0, [r2 + r4 * 2]342
+ mova m1, [r3 + r5 * 2]343
+ pmovzxbw m2, [r2 + r6]344
+ mova m3, [r3 + r7]345
+ paddw m0, m1346
+ paddw m2, m3347
+ packuswb m0, m2348
+ vpermq m0, m4, m0349
+ mova [r0 + r1 * 2], ym0350
+ vextracti32x8 [r0 + r8], m0, 1351
+%endmacro352
+353
+%macro PROCESS_ADD_PS_32x4_HBD_ALIGNED_AVX512 0354
+ mova m0, [r2]355
+ mova m1, [r2 + r4]356
+ mova m2, [r3]357
+ mova m3, [r3 + r5]358
+ paddw m0, m2359
+ paddw m1, m3360
+361
+ CLIPW2 m0, m1, m4, m5362
+ mova [r0], m0363
+ mova [r0 + r1], m1364
+365
+ mova m0, [r2 + r4 * 2]366
+ mova m1, [r2 + r6]367
+ mova m2, [r3 + r5 * 2]368
+ mova m3, [r3 + r7]369
+ paddw m0, m2370
+ paddw m1, m3371
+372
+ CLIPW2 m0, m1, m4, m5373
+ mova [r0 + r1 * 2], m0374
+ mova [r0 + r8], m1375
+%endmacro376
+377
+;-----------------------------------------------------------------------------378
+; void pixel_add_ps_32x32(pixel *dest, intptr_t destride, pixel *src0, int16_t *scr1, intptr_t srcStride0, intptr_t srcStride1)379
+;-----------------------------------------------------------------------------380
+%if HIGH_BIT_DEPTH381
+%if ARCH_X86_64382
+INIT_ZMM avx512383
+cglobal pixel_add_ps_32x32, 6, 9, 6384
+ vbroadcasti32x8 m5, [pw_pixel_max]385
+ pxor m4, m4386
+ add r4d, r4d387
+ add r5d, r5d388
+ add r1d, r1d389
+ lea r6, [r4 * 3]390
+ lea r7, [r5 * 3]391
+ lea r8, [r1 * 3]392
+%rep 7393
+ PROCESS_ADD_PS_32x4_HBD_AVX512394
+ lea r2, [r2 + r4 * 4]395
+ lea r3, [r3 + r5 * 4]396
+ lea r0, [r0 + r1 * 4]397
+%endrep398
+ PROCESS_ADD_PS_32x4_HBD_AVX512399
+ RET400
+401
+INIT_ZMM avx512402
+cglobal pixel_add_ps_32x64, 6, 9, 6403
+ vbroadcasti32x8 m5, [pw_pixel_max]404
+ pxor m4, m4405
+ add r4d, r4d406
+ add r5d, r5d407
+ add r1d, r1d408
+ lea r6, [r4 * 3]409
+ lea r7, [r5 * 3]410
+ lea r8, [r1 * 3]411
+%rep 15412
+ PROCESS_ADD_PS_32x4_HBD_AVX512413
+ lea r2, [r2 + r4 * 4]414
+ lea r3, [r3 + r5 * 4]415
+ lea r0, [r0 + r1 * 4]416
+%endrep417
+ PROCESS_ADD_PS_32x4_HBD_AVX512418
+ RET419
+420
+INIT_ZMM avx512421
+cglobal pixel_add_ps_aligned_32x32, 6, 9, 6422
+ vbroadcasti32x8 m5, [pw_pixel_max]423
+ pxor m4, m4424
+ add r4d, r4d425
+ add r5d, r5d426
+ add r1d, r1d427
+ lea r6, [r4 * 3]428
+ lea r7, [r5 * 3]429
+ lea r8, [r1 * 3]430
+%rep 7431
+ PROCESS_ADD_PS_32x4_HBD_ALIGNED_AVX512432
+ lea r2, [r2 + r4 * 4]433
+ lea r3, [r3 + r5 * 4]434
+ lea r0, [r0 + r1 * 4]435
+%endrep436
+ PROCESS_ADD_PS_32x4_HBD_ALIGNED_AVX512437
+ RET438
+439
+INIT_ZMM avx512440
+cglobal pixel_add_ps_aligned_32x64, 6, 9, 6441
+ vbroadcasti32x8 m5, [pw_pixel_max]442
+ pxor m4, m4443
+ add r4d, r4d444
+ add r5d, r5d445
+ add r1d, r1d446
+ lea r6, [r4 * 3]447
+ lea r7, [r5 * 3]448
+ lea r8, [r1 * 3]449
+%rep 15450
+ PROCESS_ADD_PS_32x4_HBD_ALIGNED_AVX512451
+ lea r2, [r2 + r4 * 4]452
+ lea r3, [r3 + r5 * 4]453
+ lea r0, [r0 + r1 * 4]454
+%endrep455
+ PROCESS_ADD_PS_32x4_HBD_ALIGNED_AVX512456
+ RET457
+%endif458
+%else459
+%if ARCH_X86_64460
+INIT_ZMM avx512461
+cglobal pixel_add_ps_32x32, 6, 9, 5462
+ add r5, r5463
+ lea r6, [r4 * 3]464
+ lea r7, [r5 * 3]465
+ lea r8, [r1 * 3]466
+ mova m4, [store_shuf1_avx512]467
+%rep 7468
+ PROCESS_ADD_PS_32x4_AVX512469
+ lea r2, [r2 + r4 * 4]470
+ lea r3, [r3 + r5 * 4]471
+ lea r0, [r0 + r1 * 4]472
+%endrep473
+ PROCESS_ADD_PS_32x4_AVX512474
+ RET475
+476
+INIT_ZMM avx512477
+cglobal pixel_add_ps_32x64, 6, 9, 5478
+ add r5, r5479
+ lea r6, [r4 * 3]480
+ lea r7, [r5 * 3]481
+ lea r8, [r1 * 3]482
+ mova m4, [store_shuf1_avx512]483
+484
+%rep 15485
+ PROCESS_ADD_PS_32x4_AVX512486
+ lea r2, [r2 + r4 * 4]487
+ lea r3, [r3 + r5 * 4]488
+ lea r0, [r0 + r1 * 4]489
+%endrep490
+ PROCESS_ADD_PS_32x4_AVX512491
+ RET492
+493
+INIT_ZMM avx512494
+cglobal pixel_add_ps_aligned_32x32, 6, 9, 5495
+ add r5, r5496
+ lea r6, [r4 * 3]497
+ lea r7, [r5 * 3]498
+ lea r8, [r1 * 3]499
+ mova m4, [store_shuf1_avx512]500
+%rep 7501
+ PROCESS_ADD_PS_32x4_ALIGNED_AVX512502
+ lea r2, [r2 + r4 * 4]503
+ lea r3, [r3 + r5 * 4]504
+ lea r0, [r0 + r1 * 4]505
+%endrep506
+ PROCESS_ADD_PS_32x4_ALIGNED_AVX512507
+ RET508
+509
+INIT_ZMM avx512510
+cglobal pixel_add_ps_aligned_32x64, 6, 9, 5511
+ add r5, r5512
+ lea r6, [r4 * 3]513
+ lea r7, [r5 * 3]514
+ lea r8, [r1 * 3]515
+ mova m4, [store_shuf1_avx512]516
+517
+%rep 15518
+ PROCESS_ADD_PS_32x4_ALIGNED_AVX512519
+ lea r2, [r2 + r4 * 4]520
+ lea r3, [r3 + r5 * 4]521
+ lea r0, [r0 + r1 * 4]522
+%endrep523
+ PROCESS_ADD_PS_32x4_ALIGNED_AVX512524
+ RET525
+%endif526
+%endif527
+;-----------------------------------------------------------------------------528
+; pixel_add_ps avx512 code end529
+;-----------------------------------------------------------------------------530
x265_2.7.tar.gz/source/common/x86/sad-a.asm -> x265_2.9.tar.gz/source/common/x86/sad-a.asm
Changed
877
1
2
lea r0, [r0 + r1]3
%endmacro4
5
-%macro SAD_W16 06
-;-----------------------------------------------------------------------------7
-; int pixel_sad_16x16( uint8_t *, intptr_t, uint8_t *, intptr_t )8
-;-----------------------------------------------------------------------------9
-cglobal pixel_sad_16x16, 4,4,810
- movu m0, [r2]11
- movu m1, [r2+r3]12
- lea r2, [r2+2*r3]13
- movu m2, [r2]14
- movu m3, [r2+r3]15
- lea r2, [r2+2*r3]16
- psadbw m0, [r0]17
- psadbw m1, [r0+r1]18
- lea r0, [r0+2*r1]19
- movu m4, [r2]20
- paddw m0, m121
- psadbw m2, [r0]22
- psadbw m3, [r0+r1]23
- lea r0, [r0+2*r1]24
- movu m5, [r2+r3]25
- lea r2, [r2+2*r3]26
- paddw m2, m327
- movu m6, [r2]28
- movu m7, [r2+r3]29
- lea r2, [r2+2*r3]30
- paddw m0, m231
- psadbw m4, [r0]32
- psadbw m5, [r0+r1]33
- lea r0, [r0+2*r1]34
- movu m1, [r2]35
- paddw m4, m536
- psadbw m6, [r0]37
- psadbw m7, [r0+r1]38
- lea r0, [r0+2*r1]39
- movu m2, [r2+r3]40
- lea r2, [r2+2*r3]41
- paddw m6, m742
- movu m3, [r2]43
- paddw m0, m444
- movu m4, [r2+r3]45
- lea r2, [r2+2*r3]46
- paddw m0, m647
- psadbw m1, [r0]48
- psadbw m2, [r0+r1]49
- lea r0, [r0+2*r1]50
- movu m5, [r2]51
- paddw m1, m252
- psadbw m3, [r0]53
- psadbw m4, [r0+r1]54
- lea r0, [r0+2*r1]55
- movu m6, [r2+r3]56
- lea r2, [r2+2*r3]57
- paddw m3, m458
- movu m7, [r2]59
- paddw m0, m160
- movu m1, [r2+r3]61
- paddw m0, m362
- psadbw m5, [r0]63
- psadbw m6, [r0+r1]64
- lea r0, [r0+2*r1]65
- paddw m5, m666
- psadbw m7, [r0]67
- psadbw m1, [r0+r1]68
- paddw m7, m169
- paddw m0, m570
- paddw m0, m771
- SAD_END_SSE272
+%macro SAD_W16 1 ; h73
+cglobal pixel_sad_16x%1, 4,474
+%ifidn cpuname, sse275
+.skip_prologue:76
+%endif77
+%assign %%i 078
+%if ARCH_X86_6479
+ lea r6, [3*r1] ; r6 results in fewer REX prefixes than r4 and both are volatile80
+ lea r5, [3*r3]81
+%rep %1/482
+ movu m1, [r2]83
+ psadbw m1, [r0]84
+ movu m3, [r2+r3]85
+ psadbw m3, [r0+r1]86
+ movu m2, [r2+2*r3]87
+ psadbw m2, [r0+2*r1]88
+ movu m4, [r2+r5]89
+ psadbw m4, [r0+r6]90
+%if %%i != %1/4-191
+ lea r2, [r2+4*r3]92
+ lea r0, [r0+4*r1]93
+%endif94
+ paddw m1, m395
+ paddw m2, m496
+ ACCUM paddw, 0, 1, %%i97
+ paddw m0, m298
+ %assign %%i %%i+199
+%endrep100
+%else ; The cost of having to save and restore registers on x86-32101
+%rep %1/2 ; nullifies the benefit of having 3*stride in registers.102
+ movu m1, [r2]103
+ psadbw m1, [r0]104
+ movu m2, [r2+r3]105
+ psadbw m2, [r0+r1]106
+%if %%i != %1/2-1107
+ lea r2, [r2+2*r3]108
+ lea r0, [r0+2*r1]109
+%endif110
+ ACCUM paddw, 0, 1, %%i111
+ paddw m0, m2112
+ %assign %%i %%i+1113
+%endrep114
+%endif115
+ SAD_END_SSE2116
+ %endmacro117
118
-;-----------------------------------------------------------------------------119
-; int pixel_sad_16x8( uint8_t *, intptr_t, uint8_t *, intptr_t )120
-;-----------------------------------------------------------------------------121
-cglobal pixel_sad_16x8, 4,4122
- movu m0, [r2]123
- movu m2, [r2+r3]124
- lea r2, [r2+2*r3]125
- movu m3, [r2]126
- movu m4, [r2+r3]127
- psadbw m0, [r0]128
- psadbw m2, [r0+r1]129
- lea r0, [r0+2*r1]130
- psadbw m3, [r0]131
- psadbw m4, [r0+r1]132
- lea r0, [r0+2*r1]133
- lea r2, [r2+2*r3]134
- paddw m0, m2135
- paddw m3, m4136
- paddw m0, m3137
- movu m1, [r2]138
- movu m2, [r2+r3]139
- lea r2, [r2+2*r3]140
- movu m3, [r2]141
- movu m4, [r2+r3]142
- psadbw m1, [r0]143
- psadbw m2, [r0+r1]144
- lea r0, [r0+2*r1]145
- psadbw m3, [r0]146
- psadbw m4, [r0+r1]147
- lea r0, [r0+2*r1]148
- lea r2, [r2+2*r3]149
- paddw m1, m2150
- paddw m3, m4151
- paddw m0, m1152
- paddw m0, m3153
- SAD_END_SSE2154
+INIT_XMM sse2155
+SAD_W16 8156
+SAD_W16 16157
+INIT_XMM sse3158
+SAD_W16 8159
+SAD_W16 16160
+INIT_XMM sse2, aligned161
+SAD_W16 8162
+SAD_W16 16163
164
+%macro SAD_Wx 0165
;-----------------------------------------------------------------------------166
; int pixel_sad_16x12( uint8_t *, intptr_t, uint8_t *, intptr_t )167
;-----------------------------------------------------------------------------168
169
%endmacro170
171
INIT_XMM sse2172
-SAD_W16173
+SAD_Wx174
INIT_XMM sse3175
-SAD_W16176
+SAD_Wx177
INIT_XMM sse2, aligned178
-SAD_W16179
+SAD_Wx180
181
%macro SAD_INC_4x8P_SSE 1182
movq m1, [r0]183
184
SAD_INC_4x8P_SSE 1185
SAD_INC_4x8P_SSE 1186
SAD_END_SSE2187
+188
+%macro SAD_W48_AVX512 3 ; w, h, d/q189
+cglobal pixel_sad_%1x%2, 4,4190
+ kxnorb k1, k1, k1191
+ kaddb k1, k1, k1192
+%assign %%i 0193
+%if ARCH_X86_64 && %2 != 4194
+ lea r6, [3*r1]195
+ lea r5, [3*r3]196
+%rep %2/4197
+ mov%3 m1, [r0]198
+ vpbroadcast%3 m1 {k1}, [r0+r1]199
+ mov%3 m3, [r2]200
+ vpbroadcast%3 m3 {k1}, [r2+r3]201
+ mov%3 m2, [r0+2*r1]202
+ vpbroadcast%3 m2 {k1}, [r0+r6]203
+ mov%3 m4, [r2+2*r3]204
+ vpbroadcast%3 m4 {k1}, [r2+r5]205
+%if %%i != %2/4-1206
+ lea r0, [r0+4*r1]207
+ lea r2, [r2+4*r3]208
+%endif209
+ psadbw m1, m3210
+ psadbw m2, m4211
+ ACCUM paddd, 0, 1, %%i212
+ paddd m0, m2213
+ %assign %%i %%i+1214
+%endrep215
+%else216
+%rep %2/2217
+ mov%3 m1, [r0]218
+ vpbroadcast%3 m1 {k1}, [r0+r1]219
+ mov%3 m2, [r2]220
+ vpbroadcast%3 m2 {k1}, [r2+r3]221
+%if %%i != %2/2-1222
+ lea r0, [r0+2*r1]223
+ lea r2, [r2+2*r3]224
+%endif225
+ psadbw m1, m2226
+ ACCUM paddd, 0, 1, %%i227
+ %assign %%i %%i+1228
+%endrep229
+%endif230
+%if %1 == 8231
+ punpckhqdq m1, m0, m0232
+ paddd m0, m1233
+%endif234
+ movd eax, m0235
+ RET236
+%endmacro237
+238
+INIT_XMM avx512239
+SAD_W48_AVX512 4, 4, d240
+SAD_W48_AVX512 4, 8, d241
+SAD_W48_AVX512 4, 16, d242
+SAD_W48_AVX512 8, 4, q243
+SAD_W48_AVX512 8, 8, q244
+SAD_W48_AVX512 8, 16, q245
+246
+%macro SAD_W16_AVX512_START 1 ; h247
+ cmp r1d, 16 ; optimized for width = 16, which has the248
+ jne pixel_sad_16x%1_sse2.skip_prologue ; rows laid out contiguously in memory249
+ lea r1, [3*r3]250
+%endmacro251
+252
+%macro SAD_W16_AVX512_END 0253
+ paddd m0, m1254
+ paddd m0, m2255
+ paddd m0, m3256
+%if mmsize == 64257
+ vextracti32x8 ym1, m0, 1258
+ paddd ym0, ym1259
+%endif260
+ vextracti128 xm1, ym0, 1261
+ paddd xmm0, xm0, xm1262
+ punpckhqdq xmm1, xmm0, xmm0263
+ paddd xmm0, xmm1264
+ movd eax, xmm0265
RET266
+%endmacro267
+268
+INIT_YMM avx512269
+cglobal pixel_sad_16x8, 4,4270
+ SAD_W16_AVX512_START 8271
+ movu xm0, [r2]272
+ vinserti128 m0, [r2+r3], 1273
+ psadbw m0, [r0+0*32]274
+ movu xm1, [r2+2*r3]275
+ vinserti128 m1, [r2+r1], 1276
+ lea r2, [r2+4*r3]277
+ psadbw m1, [r0+1*32]278
+ movu xm2, [r2]279
+ vinserti128 m2, [r2+r3], 1280
+ psadbw m2, [r0+2*32]281
+ movu xm3, [r2+2*r3]282
+ vinserti128 m3, [r2+r1], 1283
+ psadbw m3, [r0+3*32]284
+ SAD_W16_AVX512_END285
+286
+INIT_ZMM avx512287
+cglobal pixel_sad_16x16, 4,4288
+ SAD_W16_AVX512_START 16289
+ movu xm0, [r2]290
+ vinserti128 ym0, [r2+r3], 1291
+ movu xm1, [r2+4*r3]292
+ vinserti32x4 m0, [r2+2*r3], 2293
+ vinserti32x4 m1, [r2+2*r1], 2294
+ vinserti32x4 m0, [r2+r1], 3295
+ lea r2, [r2+4*r3]296
+ vinserti32x4 m1, [r2+r3], 1297
+ psadbw m0, [r0+0*64]298
+ vinserti32x4 m1, [r2+r1], 3299
+ lea r2, [r2+4*r3]300
+ psadbw m1, [r0+1*64]301
+ movu xm2, [r2]302
+ vinserti128 ym2, [r2+r3], 1303
+ movu xm3, [r2+4*r3]304
+ vinserti32x4 m2, [r2+2*r3], 2305
+ vinserti32x4 m3, [r2+2*r1], 2306
+ vinserti32x4 m2, [r2+r1], 3307
+ lea r2, [r2+4*r3]308
+ vinserti32x4 m3, [r2+r3], 1309
+ psadbw m2, [r0+2*64]310
+ vinserti32x4 m3, [r2+r1], 3311
+ psadbw m3, [r0+3*64]312
+ SAD_W16_AVX512_END313
314
;=============================================================================315
; SAD x3/x4 MMX316
317
SAD_X4_48x8_AVX2318
PIXEL_SAD_X4_END_AVX2319
RET320
+321
+;------------------------------------------------------------322
+;sad_x4 avx512 code start323
+;------------------------------------------------------------324
+%macro PROCESS_SAD_X4_64x4_AVX512 0325
+ movu m4, [r0]326
+ movu m5, [r1]327
+ movu m6, [r2]328
+ movu m7, [r3]329
+ movu m8, [r4]330
+331
+ psadbw m9, m4, m5332
+ psadbw m5, m4, m6333
+ psadbw m6, m4, m7334
+ psadbw m4, m8335
+336
+ paddd m0, m9337
+ paddd m1, m5338
+ paddd m2, m6339
+ paddd m3, m4340
+341
+ movu m4, [r0 + FENC_STRIDE]342
+ movu m5, [r1 + r5]343
+ movu m6, [r2 + r5]344
+ movu m7, [r3 + r5]345
+ movu m8, [r4 + r5]346
+347
+ psadbw m9, m4, m5348
+ psadbw m5, m4, m6349
+ psadbw m6, m4, m7350
+ psadbw m4, m8351
+ paddd m0, m9352
+ paddd m1, m5353
+ paddd m2, m6354
+ paddd m3, m4355
+356
+ movu m4, [r0 + FENC_STRIDE * 2]357
+ movu m5, [r1 + r5 * 2]358
+ movu m6, [r2 + r5 * 2]359
+ movu m7, [r3 + r5 * 2]360
+ movu m8, [r4 + r5 * 2]361
+362
+ psadbw m9, m4, m5363
+ psadbw m5, m4, m6364
+ psadbw m6, m4, m7365
+ psadbw m4, m8366
+367
+ paddd m0, m9368
+ paddd m1, m5369
+ paddd m2, m6370
+ paddd m3, m4371
+372
+ movu m4, [r0 + FENC_STRIDE * 3]373
+ movu m5, [r1 + r7]374
+ movu m6, [r2 + r7]375
+ movu m7, [r3 + r7]376
+ movu m8, [r4 + r7]377
+378
+ psadbw m9, m4, m5379
+ psadbw m5, m4, m6380
+ psadbw m6, m4, m7381
+ psadbw m4, m8382
+ paddd m0, m9383
+ paddd m1, m5384
+ paddd m2, m6385
+ paddd m3, m4386
+%endmacro387
+388
+%macro PROCESS_SAD_X4_32x4_AVX512 0389
+ movu ym4, [r0]390
+ movu ym5, [r1]391
+ movu ym6, [r2]392
+ movu ym7, [r3]393
+ movu ym8, [r4]394
+395
+ vinserti32x8 m4, [r0 + FENC_STRIDE], 1396
+ vinserti32x8 m5, [r1 + r5], 1397
+ vinserti32x8 m6, [r2 + r5], 1398
+ vinserti32x8 m7, [r3 + r5], 1399
+ vinserti32x8 m8, [r4 + r5], 1400
+401
+ psadbw m9, m4, m5402
+ psadbw m5, m4, m6403
+ psadbw m6, m4, m7404
+ psadbw m4, m8405
+406
+ paddd m0, m9407
+ paddd m1, m5408
+ paddd m2, m6409
+ paddd m3, m4410
+411
+ movu ym4, [r0 + FENC_STRIDE * 2]412
+ movu ym5, [r1 + r5 * 2]413
+ movu ym6, [r2 + r5 * 2]414
+ movu ym7, [r3 + r5 * 2]415
+ movu ym8, [r4 + r5 * 2]416
+417
+ vinserti32x8 m4, [r0 + FENC_STRIDE * 3], 1418
+ vinserti32x8 m5, [r1 + r7], 1419
+ vinserti32x8 m6, [r2 + r7], 1420
+ vinserti32x8 m7, [r3 + r7], 1421
+ vinserti32x8 m8, [r4 + r7], 1422
+423
+ psadbw m9, m4, m5424
+ psadbw m5, m4, m6425
+ psadbw m6, m4, m7426
+ psadbw m4, m8427
+428
+ paddd m0, m9429
+ paddd m1, m5430
+ paddd m2, m6431
+ paddd m3, m4432
+%endmacro433
+434
+%macro PROCESS_SAD_X4_48x4_AVX512 0435
+ movu ym4, [r0]436
+ movu ym5, [r1]437
+ movu ym6, [r2]438
+ movu ym7, [r3]439
+ movu ym8, [r4]440
+441
+ vinserti32x8 m4, [r0 + FENC_STRIDE], 1442
+ vinserti32x8 m5, [r1 + r5], 1443
+ vinserti32x8 m6, [r2 + r5], 1444
+ vinserti32x8 m7, [r3 + r5], 1445
+ vinserti32x8 m8, [r4 + r5], 1446
+447
+ psadbw m9, m4, m5448
+ psadbw m5, m4, m6449
+ psadbw m6, m4, m7450
+ psadbw m4, m8451
+452
+ paddd m0, m9453
+ paddd m1, m5454
+ paddd m2, m6455
+ paddd m3, m4456
+457
+ movu ym4, [r0 + FENC_STRIDE * 2]458
+ movu ym5, [r1 + r5 * 2]459
+ movu ym6, [r2 + r5 * 2]460
+ movu ym7, [r3 + r5 * 2]461
+ movu ym8, [r4 + r5 * 2]462
+463
+ vinserti32x8 m4, [r0 + FENC_STRIDE * 3], 1464
+ vinserti32x8 m5, [r1 + r7], 1465
+ vinserti32x8 m6, [r2 + r7], 1466
+ vinserti32x8 m7, [r3 + r7], 1467
+ vinserti32x8 m8, [r4 + r7], 1468
+469
+ psadbw m9, m4, m5470
+ psadbw m5, m4, m6471
+ psadbw m6, m4, m7472
+ psadbw m4, m8473
+474
+ paddd m0, m9475
+ paddd m1, m5476
+ paddd m2, m6477
+ paddd m3, m4478
+479
+ movu xm4, [r0 + mmsize/2]480
+ movu xm5, [r1 + mmsize/2]481
+ movu xm6, [r2 + mmsize/2]482
+ movu xm7, [r3 + mmsize/2]483
+ movu xm8, [r4 + mmsize/2]484
+ vinserti32x4 m4, [r0 + FENC_STRIDE + mmsize/2], 1485
+ vinserti32x4 m5, [r1 + r5 + mmsize/2], 1486
+ vinserti32x4 m6, [r2 + r5 + mmsize/2], 1487
+ vinserti32x4 m7, [r3 + r5 + mmsize/2], 1488
+ vinserti32x4 m8, [r4 + r5 + mmsize/2], 1489
+490
+ vinserti32x4 m4, [r0 + FENC_STRIDE * 2 + mmsize/2], 2491
+ vinserti32x4 m5, [r1 + r5 * 2 + mmsize/2], 2492
+ vinserti32x4 m6, [r2 + r5 * 2 + mmsize/2], 2493
+ vinserti32x4 m7, [r3 + r5 * 2 + mmsize/2], 2494
+ vinserti32x4 m8, [r4 + r5 * 2 + mmsize/2], 2495
+ vinserti32x4 m4, [r0 + FENC_STRIDE * 3 + mmsize/2], 3496
+ vinserti32x4 m5, [r1 + r7 + mmsize/2], 3497
+ vinserti32x4 m6, [r2 + r7 + mmsize/2], 3498
+ vinserti32x4 m7, [r3 + r7 + mmsize/2], 3499
+ vinserti32x4 m8, [r4 + r7 + mmsize/2], 3500
+501
+ psadbw m9, m4, m5502
+ psadbw m5, m4, m6503
+ psadbw m6, m4, m7504
+ psadbw m4, m8505
+ paddd m0, m9506
+ paddd m1, m5507
+ paddd m2, m6508
+ paddd m3, m4509
+%endmacro510
+511
+%macro PIXEL_SAD_X4_END_AVX512 0512
+ vextracti32x8 ym4, m0, 1513
+ vextracti32x8 ym5, m1, 1514
+ vextracti32x8 ym6, m2, 1515
+ vextracti32x8 ym7, m3, 1516
+ paddd ym0, ym4517
+ paddd ym1, ym5518
+ paddd ym2, ym6519
+ paddd ym3, ym7520
+ vextracti64x2 xm4, m0, 1521
+ vextracti64x2 xm5, m1, 1522
+ vextracti64x2 xm6, m2, 1523
+ vextracti64x2 xm7, m3, 1524
+ paddd xm0, xm4525
+ paddd xm1, xm5526
+ paddd xm2, xm6527
+ paddd xm3, xm7528
+ pshufd xm4, xm0, 2529
+ pshufd xm5, xm1, 2530
+ pshufd xm6, xm2, 2531
+ pshufd xm7, xm3, 2532
+ paddd xm0, xm4533
+ paddd xm1, xm5534
+ paddd xm2, xm6535
+ paddd xm3, xm7536
+ movd [r6 + 0], xm0537
+ movd [r6 + 4], xm1538
+ movd [r6 + 8], xm2539
+ movd [r6 + 12], xm3540
+%endmacro541
+542
+%macro SAD_X4_AVX512 2543
+INIT_ZMM avx512544
+cglobal pixel_sad_x4_%1x%2, 7,8,10545
+ pxor m0, m0546
+ pxor m1, m1547
+ pxor m2, m2548
+ pxor m3, m3549
+ lea r7, [r5 * 3]550
+551
+%rep %2/4 - 1552
+ PROCESS_SAD_X4_%1x4_AVX512553
+ add r0, FENC_STRIDE * 4554
+ lea r1, [r1 + r5 * 4]555
+ lea r2, [r2 + r5 * 4]556
+ lea r3, [r3 + r5 * 4]557
+ lea r4, [r4 + r5 * 4]558
+%endrep559
+ PROCESS_SAD_X4_%1x4_AVX512560
+ PIXEL_SAD_X4_END_AVX512561
+ RET562
+%endmacro563
+564
+SAD_X4_AVX512 64, 64565
+SAD_X4_AVX512 64, 48566
+SAD_X4_AVX512 64, 32567
+SAD_X4_AVX512 64, 16568
+SAD_X4_AVX512 32, 64569
+SAD_X4_AVX512 32, 32570
+SAD_X4_AVX512 32, 24571
+SAD_X4_AVX512 32, 16572
+SAD_X4_AVX512 32, 8573
+SAD_X4_AVX512 48, 64574
+;------------------------------------------------------------575
+;sad_x4 avx512 code end576
+;------------------------------------------------------------577
%endif578
579
INIT_XMM sse2580
581
RET582
%endif583
584
+;------------------------------------------------------------585
+;sad_x3 avx512 code start586
+;------------------------------------------------------------587
+%macro PROCESS_SAD_X3_64x4_AVX512 0588
+ movu m3, [r0]589
+ movu m4, [r1]590
+ movu m5, [r2]591
+ movu m6, [r3]592
+593
+ psadbw m7, m3, m4594
+ psadbw m4, m3, m5595
+ psadbw m3, m6596
+597
+ paddd m0, m7598
+ paddd m1, m4599
+ paddd m2, m3600
+601
+ movu m3, [r0 + FENC_STRIDE]602
+ movu m4, [r1 + r4]603
+ movu m5, [r2 + r4]604
+ movu m6, [r3 + r4]605
+606
+ psadbw m7, m3, m4607
+ psadbw m4, m3, m5608
+ psadbw m3, m6609
+610
+ paddd m0, m7611
+ paddd m1, m4612
+ paddd m2, m3613
+614
+ movu m3, [r0 + FENC_STRIDE * 2]615
+ movu m4, [r1 + r4 * 2]616
+ movu m5, [r2 + r4 * 2]617
+ movu m6, [r3 + r4 * 2]618
+619
+ psadbw m7, m3, m4620
+ psadbw m4, m3, m5621
+ psadbw m3, m6622
+623
+ paddd m0, m7624
+ paddd m1, m4625
+ paddd m2, m3626
+627
+ movu m3, [r0 + FENC_STRIDE * 3]628
+ movu m4, [r1 + r6]629
+ movu m5, [r2 + r6]630
+ movu m6, [r3 + r6]631
+632
+ psadbw m7, m3, m4633
+ psadbw m4, m3, m5634
+ psadbw m3, m6635
+636
+ paddd m0, m7637
+ paddd m1, m4638
+ paddd m2, m3639
+%endmacro640
+641
+%macro PROCESS_SAD_X3_32x4_AVX512 0642
+ movu ym3, [r0]643
+ movu ym4, [r1]644
+ movu ym5, [r2]645
+ movu ym6, [r3]646
+ vinserti32x8 m3, [r0 + FENC_STRIDE], 1647
+ vinserti32x8 m4, [r1 + r4], 1648
+ vinserti32x8 m5, [r2 + r4], 1649
+ vinserti32x8 m6, [r3 + r4], 1650
+651
+ psadbw m7, m3, m4652
+ psadbw m4, m3, m5653
+ psadbw m3, m6654
+655
+ paddd m0, m7656
+ paddd m1, m4657
+ paddd m2, m3658
+659
+ movu ym3, [r0 + FENC_STRIDE * 2]660
+ movu ym4, [r1 + r4 * 2]661
+ movu ym5, [r2 + r4 * 2]662
+ movu ym6, [r3 + r4 * 2]663
+ vinserti32x8 m3, [r0 + FENC_STRIDE * 3], 1664
+ vinserti32x8 m4, [r1 + r6], 1665
+ vinserti32x8 m5, [r2 + r6], 1666
+ vinserti32x8 m6, [r3 + r6], 1667
+668
+ psadbw m7, m3, m4669
+ psadbw m4, m3, m5670
+ psadbw m3, m6671
+672
+ paddd m0, m7673
+ paddd m1, m4674
+ paddd m2, m3675
+%endmacro676
+677
+%macro PROCESS_SAD_X3_48x4_AVX512 0678
+ movu ym3, [r0]679
+ movu ym4, [r1]680
+ movu ym5, [r2]681
+ movu ym6, [r3]682
+ vinserti32x8 m3, [r0 + FENC_STRIDE], 1683
+ vinserti32x8 m4, [r1 + r4], 1684
+ vinserti32x8 m5, [r2 + r4], 1685
+ vinserti32x8 m6, [r3 + r4], 1686
+687
+ psadbw m7, m3, m4688
+ psadbw m4, m3, m5689
+ psadbw m3, m6690
+691
+ paddd m0, m7692
+ paddd m1, m4693
+ paddd m2, m3694
+695
+ movu ym3, [r0 + FENC_STRIDE * 2]696
+ movu ym4, [r1 + r4 * 2]697
+ movu ym5, [r2 + r4 * 2]698
+ movu ym6, [r3 + r4 * 2]699
+ vinserti32x8 m3, [r0 + FENC_STRIDE * 3], 1700
+ vinserti32x8 m4, [r1 + r6], 1701
+ vinserti32x8 m5, [r2 + r6], 1702
+ vinserti32x8 m6, [r3 + r6], 1703
+704
+ psadbw m7, m3, m4705
+ psadbw m4, m3, m5706
+ psadbw m3, m6707
+708
+ paddd m0, m7709
+ paddd m1, m4710
+ paddd m2, m3711
+712
+ movu xm3, [r0 + mmsize/2]713
+ movu xm4, [r1 + mmsize/2]714
+ movu xm5, [r2 + mmsize/2]715
+ movu xm6, [r3 + mmsize/2]716
+ vinserti32x4 m3, [r0 + FENC_STRIDE + mmsize/2], 1717
+ vinserti32x4 m4, [r1 + r4 + mmsize/2], 1718
+ vinserti32x4 m5, [r2 + r4 + mmsize/2], 1719
+ vinserti32x4 m6, [r3 + r4 + mmsize/2], 1720
+721
+ vinserti32x4 m3, [r0 + 2 * FENC_STRIDE + mmsize/2], 2722
+ vinserti32x4 m4, [r1 + 2 * r4 + mmsize/2], 2723
+ vinserti32x4 m5, [r2 + 2 * r4 + mmsize/2], 2724
+ vinserti32x4 m6, [r3 + 2 * r4 + mmsize/2], 2725
+ vinserti32x4 m3, [r0 + 3 * FENC_STRIDE + mmsize/2], 3726
+ vinserti32x4 m4, [r1 + r6 + mmsize/2], 3727
+ vinserti32x4 m5, [r2 + r6 + mmsize/2], 3728
+ vinserti32x4 m6, [r3 + r6 + mmsize/2], 3729
+730
+ psadbw m7, m3, m4731
+ psadbw m4, m3, m5732
+ psadbw m3, m6733
+ paddd m0, m7734
+ paddd m1, m4735
+ paddd m2, m3736
+%endmacro737
+738
+%macro PIXEL_SAD_X3_END_AVX512 0739
+ vextracti32x8 ym3, m0, 1740
+ vextracti32x8 ym4, m1, 1741
+ vextracti32x8 ym5, m2, 1742
+ paddd ym0, ym3743
+ paddd ym1, ym4744
+ paddd ym2, ym5745
+ vextracti64x2 xm3, m0, 1746
+ vextracti64x2 xm4, m1, 1747
+ vextracti64x2 xm5, m2, 1748
+ paddd xm0, xm3749
+ paddd xm1, xm4750
+ paddd xm2, xm5751
+ pshufd xm3, xm0, 2752
+ pshufd xm4, xm1, 2753
+ pshufd xm5, xm2, 2754
+ paddd xm0, xm3755
+ paddd xm1, xm4756
+ paddd xm2, xm5757
+ movd [r5 + 0], xm0758
+ movd [r5 + 4], xm1759
+ movd [r5 + 8], xm2760
+%endmacro761
+762
+%macro SAD_X3_AVX512 2763
+INIT_ZMM avx512764
+cglobal pixel_sad_x3_%1x%2, 6,7,8765
+ pxor m0, m0766
+ pxor m1, m1767
+ pxor m2, m2768
+ lea r6, [r4 * 3]769
+770
+%rep %2/4 - 1771
+ PROCESS_SAD_X3_%1x4_AVX512772
+ add r0, FENC_STRIDE * 4773
+ lea r1, [r1 + r4 * 4]774
+ lea r2, [r2 + r4 * 4]775
+ lea r3, [r3 + r4 * 4]776
+%endrep777
+ PROCESS_SAD_X3_%1x4_AVX512778
+ PIXEL_SAD_X3_END_AVX512779
+ RET780
+%endmacro781
+782
+SAD_X3_AVX512 64, 64783
+SAD_X3_AVX512 64, 48784
+SAD_X3_AVX512 64, 32785
+SAD_X3_AVX512 64, 16786
+SAD_X3_AVX512 32, 64787
+SAD_X3_AVX512 32, 32788
+SAD_X3_AVX512 32, 24789
+SAD_X3_AVX512 32, 16790
+SAD_X3_AVX512 32, 8791
+SAD_X3_AVX512 48, 64792
+;------------------------------------------------------------793
+;sad_x3 avx512 code end794
+;------------------------------------------------------------795
+796
INIT_YMM avx2797
cglobal pixel_sad_x4_8x8, 7,7,5798
xorps m0, m0799
800
movd eax, xm0801
RET802
803
+%macro PROCESS_SAD_64x4_AVX512 0804
+ movu m1, [r0]805
+ movu m2, [r2]806
+ movu m3, [r0 + r1]807
+ movu m4, [r2 + r3]808
+ psadbw m1, m2809
+ psadbw m3, m4810
+ paddd m0, m1811
+ paddd m0, m3812
+ movu m1, [r0 + 2 * r1]813
+ movu m2, [r2 + 2 * r3]814
+ movu m3, [r0 + r5]815
+ movu m4, [r2 + r6]816
+ psadbw m1, m2817
+ psadbw m3, m4818
+ paddd m0, m1819
+ paddd m0, m3820
+%endmacro821
+822
+%macro PROCESS_SAD_32x4_AVX512 0823
+ movu ym1, [r0]824
+ movu ym2, [r2]825
+ movu ym3, [r0 + 2 * r1]826
+ movu ym4, [r2 + 2 * r3]827
+ vinserti32x8 m1, [r0 + r1], 1828
+ vinserti32x8 m2, [r2 + r3], 1829
+ vinserti32x8 m3, [r0 + r5], 1830
+ vinserti32x8 m4, [r2 + r6], 1831
+832
+ psadbw m1, m2833
+ psadbw m3, m4834
+ paddd m0, m1835
+ paddd m0, m3836
+%endmacro837
+838
+%macro PROCESS_SAD_AVX512_END 0839
+ vextracti32x8 ym1, m0, 1840
+ paddd ym0, ym1841
+ vextracti64x2 xm1, m0, 1842
+ paddd xm0, xm1843
+ pshufd xm1, xm0, 2844
+ paddd xm0, xm1845
+ movd eax, xm0846
+%endmacro847
+;-----------------------------------------------------------------------------848
+; int pixel_sad_64x%1( uint8_t *, intptr_t, uint8_t *, intptr_t )849
+;-----------------------------------------------------------------------------850
+%macro SAD_MxN_AVX512 2851
+INIT_ZMM avx512852
+cglobal pixel_sad_%1x%2, 4, 7, 5853
+ pxor m0, m0854
+ lea r5, [3 * r1]855
+ lea r6, [3 * r3]856
+857
+%rep %2/4 - 1858
+ PROCESS_SAD_%1x4_AVX512859
+ lea r2, [r2 + 4 * r3]860
+ lea r0, [r0 + 4 * r1]861
+%endrep862
+ PROCESS_SAD_%1x4_AVX512863
+ PROCESS_SAD_AVX512_END864
+ RET865
+%endmacro866
+867
+SAD_MxN_AVX512 64, 16868
+SAD_MxN_AVX512 64, 32869
+SAD_MxN_AVX512 64, 48870
+SAD_MxN_AVX512 64, 64871
+SAD_MxN_AVX512 32, 8872
+SAD_MxN_AVX512 32, 16873
+SAD_MxN_AVX512 32, 24874
+SAD_MxN_AVX512 32, 32875
+SAD_MxN_AVX512 32, 64876
%endif877
x265_2.7.tar.gz/source/common/x86/sad16-a.asm -> x265_2.9.tar.gz/source/common/x86/sad16-a.asm
Changed
2819
1
2
SAD_12 12, 163
4
5
+%macro PROCESS_SAD_64x8_AVX512 06
+ movu m1, [r2]7
+ movu m2, [r2 + mmsize]8
+ movu m3, [r2 + r3]9
+ movu m4, [r2 + r3 + mmsize]10
+ psubw m1, [r0]11
+ psubw m2, [r0 + mmsize]12
+ psubw m3, [r0 + r1]13
+ psubw m4, [r0 + r1 + mmsize]14
+ pabsw m1, m115
+ pabsw m2, m216
+ pabsw m3, m317
+ pabsw m4, m418
+ paddw m1, m219
+ paddw m3, m420
+ paddw m5, m1, m321
+22
+ movu m1, [r2 + 2 * r3]23
+ movu m2, [r2 + 2 * r3 + mmsize]24
+ movu m3, [r2 + r5]25
+ movu m4, [r2 + r5 + mmsize]26
+ psubw m1, [r0 + 2 * r1]27
+ psubw m2, [r0 + 2 * r1 + mmsize]28
+ psubw m3, [r0 + r4]29
+ psubw m4, [r0 + r4 + mmsize]30
+ pabsw m1, m131
+ pabsw m2, m232
+ pabsw m3, m333
+ pabsw m4, m434
+ paddw m1, m235
+ paddw m3, m436
+ paddw m1, m337
+38
+ lea r0, [r0 + 4 * r1]39
+ lea r2, [r2 + 4 * r3]40
+41
+ pmaddwd m5, m642
+ paddd m0, m543
+ pmaddwd m1, m644
+ paddd m0, m145
+46
+ movu m1, [r2]47
+ movu m2, [r2 + mmsize]48
+ movu m3, [r2 + r3]49
+ movu m4, [r2 + r3 + mmsize]50
+ psubw m1, [r0]51
+ psubw m2, [r0 + mmsize]52
+ psubw m3, [r0 + r1]53
+ psubw m4, [r0 + r1 + mmsize]54
+ pabsw m1, m155
+ pabsw m2, m256
+ pabsw m3, m357
+ pabsw m4, m458
+ paddw m1, m259
+ paddw m3, m460
+ paddw m5, m1, m361
+62
+ movu m1, [r2 + 2 * r3]63
+ movu m2, [r2 + 2 * r3 + mmsize]64
+ movu m3, [r2 + r5]65
+ movu m4, [r2 + r5 + mmsize]66
+ psubw m1, [r0 + 2 * r1]67
+ psubw m2, [r0 + 2 * r1 + mmsize]68
+ psubw m3, [r0 + r4]69
+ psubw m4, [r0 + r4 + mmsize]70
+ pabsw m1, m171
+ pabsw m2, m272
+ pabsw m3, m373
+ pabsw m4, m474
+ paddw m1, m275
+ paddw m3, m476
+ paddw m1, m377
+78
+ pmaddwd m5, m679
+ paddd m0, m580
+ pmaddwd m1, m681
+ paddd m0, m182
+%endmacro83
+84
+85
+%macro PROCESS_SAD_32x8_AVX512 086
+ movu m1, [r2]87
+ movu m2, [r2 + r3]88
+ movu m3, [r2 + 2 * r3]89
+ movu m4, [r2 + r5]90
+ psubw m1, [r0]91
+ psubw m2, [r0 + r1]92
+ psubw m3, [r0 + 2 * r1]93
+ psubw m4, [r0 + r4]94
+ pabsw m1, m195
+ pabsw m2, m296
+ pabsw m3, m397
+ pabsw m4, m498
+ paddw m1, m299
+ paddw m3, m4100
+ paddw m5, m1, m3101
+102
+ lea r0, [r0 + 4 * r1]103
+ lea r2, [r2 + 4 * r3]104
+105
+ movu m1, [r2]106
+ movu m2, [r2 + r3]107
+ movu m3, [r2 + 2 * r3]108
+ movu m4, [r2 + r5]109
+ psubw m1, [r0]110
+ psubw m2, [r0 + r1]111
+ psubw m3, [r0 + 2 * r1]112
+ psubw m4, [r0 + r4]113
+ pabsw m1, m1114
+ pabsw m2, m2115
+ pabsw m3, m3116
+ pabsw m4, m4117
+ paddw m1, m2118
+ paddw m3, m4119
+ paddw m1, m3120
+121
+ pmaddwd m5, m6122
+ paddd m0, m5123
+ pmaddwd m1, m6124
+ paddd m0, m1125
+%endmacro126
+127
+%macro PROCESS_SAD_16x8_AVX512 0128
+ movu ym1, [r2]129
+ vinserti64x4 m1, [r2 + r3], 1130
+ movu ym2, [r2 + 2 * r3]131
+ vinserti64x4 m2, [r2 + r5], 1132
+ movu ym3, [r0]133
+ vinserti64x4 m3, [r0 + r1], 1134
+ movu ym4, [r0 + 2 * r1]135
+ vinserti64x4 m4, [r0 + r4], 1136
+137
+ psubw m1, m3138
+ psubw m2, m4139
+ pabsw m1, m1140
+ pabsw m2, m2141
+ paddw m5, m1, m2142
+143
+ lea r0, [r0 + 4 * r1]144
+ lea r2, [r2 + 4 * r3]145
+146
+ movu ym1, [r2]147
+ vinserti64x4 m1, [r2 + r3], 1148
+ movu ym2, [r2 + 2 * r3]149
+ vinserti64x4 m2, [r2 + r5], 1150
+ movu ym3, [r0]151
+ vinserti64x4 m3, [r0 + r1], 1152
+ movu ym4, [r0 + 2 * r1]153
+ vinserti64x4 m4, [r0 + r4], 1154
+155
+ psubw m1, m3156
+ psubw m2, m4157
+ pabsw m1, m1158
+ pabsw m2, m2159
+ paddw m1, m2160
+161
+ pmaddwd m5, m6162
+ paddd m0, m5163
+ pmaddwd m1, m6164
+ paddd m0, m1165
+%endmacro166
+167
+%macro PROCESS_SAD_AVX512_END 0168
+ vextracti32x8 ym1, m0, 1169
+ paddd ym0, ym1170
+ vextracti64x2 xm1, m0, 1171
+ paddd xm0, xm1172
+ pshufd xm1, xm0, 00001110b173
+ paddd xm0, xm1174
+ pshufd xm1, xm0, 00000001b175
+ paddd xm0, xm1176
+ movd eax, xm0177
+%endmacro178
+179
+;-----------------------------------------------------------------------------180
+; int pixel_sad_64x%1( uint16_t *, intptr_t, uint16_t *, intptr_t )181
+;-----------------------------------------------------------------------------182
+%if ARCH_X86_64183
+INIT_ZMM avx512184
+cglobal pixel_sad_64x16, 4,6,7185
+ pxor m0, m0186
+187
+ vbroadcasti32x8 m6, [pw_1]188
+189
+ add r3d, r3d190
+ add r1d, r1d191
+ lea r4d, [r1 * 3]192
+ lea r5d, [r3 * 3]193
+194
+ PROCESS_SAD_64x8_AVX512195
+ lea r2, [r2 + 4 * r3]196
+ lea r0, [r0 + 4 * r1]197
+ PROCESS_SAD_64x8_AVX512198
+ PROCESS_SAD_AVX512_END199
+ RET200
+201
+INIT_ZMM avx512202
+cglobal pixel_sad_64x32, 4,6,7203
+ pxor m0, m0204
+205
+ vbroadcasti32x8 m6, [pw_1]206
+207
+ add r3d, r3d208
+ add r1d, r1d209
+ lea r4d, [r1 * 3]210
+ lea r5d, [r3 * 3]211
+212
+ PROCESS_SAD_64x8_AVX512213
+ lea r2, [r2 + 4 * r3]214
+ lea r0, [r0 + 4 * r1]215
+ PROCESS_SAD_64x8_AVX512216
+ lea r2, [r2 + 4 * r3]217
+ lea r0, [r0 + 4 * r1]218
+ PROCESS_SAD_64x8_AVX512219
+ lea r2, [r2 + 4 * r3]220
+ lea r0, [r0 + 4 * r1]221
+ PROCESS_SAD_64x8_AVX512222
+ PROCESS_SAD_AVX512_END223
+ RET224
+225
+INIT_ZMM avx512226
+cglobal pixel_sad_64x48, 4,6,7227
+ pxor m0, m0228
+229
+ vbroadcasti32x8 m6, [pw_1]230
+231
+ add r3d, r3d232
+ add r1d, r1d233
+ lea r4d, [r1 * 3]234
+ lea r5d, [r3 * 3]235
+236
+ PROCESS_SAD_64x8_AVX512237
+ lea r2, [r2 + 4 * r3]238
+ lea r0, [r0 + 4 * r1]239
+ PROCESS_SAD_64x8_AVX512240
+ lea r2, [r2 + 4 * r3]241
+ lea r0, [r0 + 4 * r1]242
+ PROCESS_SAD_64x8_AVX512243
+ lea r2, [r2 + 4 * r3]244
+ lea r0, [r0 + 4 * r1]245
+ PROCESS_SAD_64x8_AVX512246
+ lea r2, [r2 + 4 * r3]247
+ lea r0, [r0 + 4 * r1]248
+ PROCESS_SAD_64x8_AVX512249
+ lea r2, [r2 + 4 * r3]250
+ lea r0, [r0 + 4 * r1]251
+ PROCESS_SAD_64x8_AVX512252
+ PROCESS_SAD_AVX512_END253
+ RET254
+255
+INIT_ZMM avx512256
+cglobal pixel_sad_64x64, 4,6,7257
+ pxor m0, m0258
+259
+ vbroadcasti32x8 m6, [pw_1]260
+261
+ add r3d, r3d262
+ add r1d, r1d263
+ lea r4d, [r1 * 3]264
+ lea r5d, [r3 * 3]265
+266
+ PROCESS_SAD_64x8_AVX512267
+ lea r2, [r2 + 4 * r3]268
+ lea r0, [r0 + 4 * r1]269
+ PROCESS_SAD_64x8_AVX512270
+ lea r2, [r2 + 4 * r3]271
+ lea r0, [r0 + 4 * r1]272
+ PROCESS_SAD_64x8_AVX512273
+ lea r2, [r2 + 4 * r3]274
+ lea r0, [r0 + 4 * r1]275
+ PROCESS_SAD_64x8_AVX512276
+ lea r2, [r2 + 4 * r3]277
+ lea r0, [r0 + 4 * r1]278
+ PROCESS_SAD_64x8_AVX512279
+ lea r2, [r2 + 4 * r3]280
+ lea r0, [r0 + 4 * r1]281
+ PROCESS_SAD_64x8_AVX512282
+ lea r2, [r2 + 4 * r3]283
+ lea r0, [r0 + 4 * r1]284
+ PROCESS_SAD_64x8_AVX512285
+ lea r2, [r2 + 4 * r3]286
+ lea r0, [r0 + 4 * r1]287
+ PROCESS_SAD_64x8_AVX512288
+ PROCESS_SAD_AVX512_END289
+ RET290
+%endif291
+292
+;-----------------------------------------------------------------------------293
+; int pixel_sad_32x%1( uint16_t *, intptr_t, uint16_t *, intptr_t )294
+;-----------------------------------------------------------------------------295
+%if ARCH_X86_64296
+INIT_ZMM avx512297
+cglobal pixel_sad_32x8, 4,6,7298
+ pxor m0, m0299
+300
+ vbroadcasti32x8 m6, [pw_1]301
+302
+ add r3d, r3d303
+ add r1d, r1d304
+ lea r4d, [r1 * 3]305
+ lea r5d, [r3 * 3]306
+307
+ PROCESS_SAD_32x8_AVX512308
+ PROCESS_SAD_AVX512_END309
+ RET310
+311
+312
+INIT_ZMM avx512313
+cglobal pixel_sad_32x16, 4,6,7314
+ pxor m0, m0315
+316
+ vbroadcasti32x8 m6, [pw_1]317
+318
+ add r3d, r3d319
+ add r1d, r1d320
+ lea r4d, [r1 * 3]321
+ lea r5d, [r3 * 3]322
+323
+ PROCESS_SAD_32x8_AVX512324
+ lea r2, [r2 + 4 * r3]325
+ lea r0, [r0 + 4 * r1]326
+ PROCESS_SAD_32x8_AVX512327
+ PROCESS_SAD_AVX512_END328
+ RET329
+330
+INIT_ZMM avx512331
+cglobal pixel_sad_32x24, 4,6,7332
+ pxor m0, m0333
+334
+ vbroadcasti32x8 m6, [pw_1]335
+336
+ add r3d, r3d337
+ add r1d, r1d338
+ lea r4d, [r1 * 3]339
+ lea r5d, [r3 * 3]340
+341
+ PROCESS_SAD_32x8_AVX512342
+ lea r2, [r2 + 4 * r3]343
+ lea r0, [r0 + 4 * r1]344
+ PROCESS_SAD_32x8_AVX512345
+ lea r2, [r2 + 4 * r3]346
+ lea r0, [r0 + 4 * r1]347
+ PROCESS_SAD_32x8_AVX512348
+ PROCESS_SAD_AVX512_END349
+ RET350
+351
+INIT_ZMM avx512352
+cglobal pixel_sad_32x32, 4,6,7353
+ pxor m0, m0354
+355
+ vbroadcasti32x8 m6, [pw_1]356
+357
+ add r3d, r3d358
+ add r1d, r1d359
+ lea r4d, [r1 * 3]360
+ lea r5d, [r3 * 3]361
+362
+ PROCESS_SAD_32x8_AVX512363
+ lea r2, [r2 + 4 * r3]364
+ lea r0, [r0 + 4 * r1]365
+ PROCESS_SAD_32x8_AVX512366
+ lea r2, [r2 + 4 * r3]367
+ lea r0, [r0 + 4 * r1]368
+ PROCESS_SAD_32x8_AVX512369
+ lea r2, [r2 + 4 * r3]370
+ lea r0, [r0 + 4 * r1]371
+ PROCESS_SAD_32x8_AVX512372
+ PROCESS_SAD_AVX512_END373
+ RET374
+375
+INIT_ZMM avx512376
+cglobal pixel_sad_32x64, 4,6,7377
+ pxor m0, m0378
+379
+ vbroadcasti32x8 m6, [pw_1]380
+381
+ add r3d, r3d382
+ add r1d, r1d383
+ lea r4d, [r1 * 3]384
+ lea r5d, [r3 * 3]385
+386
+ PROCESS_SAD_32x8_AVX512387
+ lea r2, [r2 + 4 * r3]388
+ lea r0, [r0 + 4 * r1]389
+ PROCESS_SAD_32x8_AVX512390
+ lea r2, [r2 + 4 * r3]391
+ lea r0, [r0 + 4 * r1]392
+ PROCESS_SAD_32x8_AVX512393
+ lea r2, [r2 + 4 * r3]394
+ lea r0, [r0 + 4 * r1]395
+ PROCESS_SAD_32x8_AVX512396
+ lea r2, [r2 + 4 * r3]397
+ lea r0, [r0 + 4 * r1]398
+ PROCESS_SAD_32x8_AVX512399
+ lea r2, [r2 + 4 * r3]400
+ lea r0, [r0 + 4 * r1]401
+ PROCESS_SAD_32x8_AVX512402
+ lea r2, [r2 + 4 * r3]403
+ lea r0, [r0 + 4 * r1]404
+ PROCESS_SAD_32x8_AVX512405
+ lea r2, [r2 + 4 * r3]406
+ lea r0, [r0 + 4 * r1]407
+ PROCESS_SAD_32x8_AVX512408
+ PROCESS_SAD_AVX512_END409
+ RET410
+%endif411
+412
+;-----------------------------------------------------------------------------413
+; int pixel_sad_16x%1( uint16_t *, intptr_t, uint16_t *, intptr_t )414
+;-----------------------------------------------------------------------------415
+%if ARCH_X86_64416
+INIT_ZMM avx512417
+cglobal pixel_sad_16x32, 4,6,7418
+ pxor m0, m0419
+420
+ vbroadcasti32x8 m6, [pw_1]421
+422
+ add r3d, r3d423
+ add r1d, r1d424
+ lea r4d, [r1 * 3]425
+ lea r5d, [r3 * 3]426
+427
+ %rep 3428
+ PROCESS_SAD_16x8_AVX512429
+ lea r2, [r2 + 4 * r3]430
+ lea r0, [r0 + 4 * r1]431
+ %endrep432
+ PROCESS_SAD_16x8_AVX512433
+ PROCESS_SAD_AVX512_END434
+ RET435
+436
+INIT_ZMM avx512437
+cglobal pixel_sad_16x64, 4,6,7438
+ pxor m0, m0439
+440
+ vbroadcasti32x8 m6, [pw_1]441
+442
+ add r3d, r3d443
+ add r1d, r1d444
+ lea r4d, [r1 * 3]445
+ lea r5d, [r3 * 3]446
+447
+ %rep 7448
+ PROCESS_SAD_16x8_AVX512449
+ lea r2, [r2 + 4 * r3]450
+ lea r0, [r0 + 4 * r1]451
+ %endrep452
+ PROCESS_SAD_16x8_AVX512453
+ PROCESS_SAD_AVX512_END454
+ RET455
+%endif456
+457
+;-----------------------------------------------------------------------------458
+; int pixel_sad_48x64( uint16_t *, intptr_t, uint16_t *, intptr_t )459
+;-----------------------------------------------------------------------------460
+%if ARCH_X86_64461
+INIT_ZMM avx512462
+cglobal pixel_sad_48x64, 4, 7, 9463
+ pxor m0, m0464
+ mov r6d, 64/8465
+466
+ vbroadcasti32x8 m8, [pw_1]467
+468
+ add r3d, r3d469
+ add r1d, r1d470
+ lea r4d, [r1 * 3]471
+ lea r5d, [r3 * 3]472
+.loop:473
+ movu m1, [r2]474
+ movu m2, [r2 + r3]475
+ movu ym3, [r2 + mmsize]476
+ vinserti32x8 m3, [r2 + r3 + mmsize], 1477
+ movu m4, [r0]478
+ movu m5, [r0 + r1]479
+ movu ym6, [r0 + mmsize]480
+ vinserti32x8 m6, [r0 + r1 + mmsize], 1481
+482
+ psubw m1, m4483
+ psubw m2, m5484
+ psubw m3, m6485
+ pabsw m1, m1486
+ pabsw m2, m2487
+ pabsw m3, m3488
+ paddw m1, m2489
+ paddw m7, m3, m1490
+491
+ movu m1, [r2 + 2 * r3]492
+ movu m2, [r2 + r5]493
+ movu ym3, [r2 + 2 * r3 + mmsize]494
+ vinserti32x8 m3, [r2 + r5 + mmsize], 1495
+ movu m4, [r0 + 2 * r1]496
+ movu m5, [r0 + r4]497
+ movu ym6, [r0 + 2 * r1 + mmsize]498
+ vinserti32x8 m6, [r0 + r4 + mmsize], 1499
+ psubw m1, m4500
+ psubw m2, m5501
+ psubw m3, m6502
+ pabsw m1, m1503
+ pabsw m2, m2504
+ pabsw m3, m3505
+ paddw m1, m2506
+ paddw m1, m3507
+508
+ pmaddwd m7, m8509
+ paddd m0, m7510
+ pmaddwd m1, m8511
+ paddd m0, m1512
+ lea r0, [r0 + 4 * r1]513
+ lea r2, [r2 + 4 * r3]514
+515
+ movu m1, [r2]516
+ movu m2, [r2 + r3]517
+ movu ym3, [r2 + mmsize]518
+ vinserti32x8 m3, [r2 + r3 + mmsize], 1519
+ movu m4, [r0]520
+ movu m5, [r0 + r1]521
+ movu ym6, [r0 + mmsize]522
+ vinserti32x8 m6, [r0 + r1 + mmsize], 1523
+524
+ psubw m1, m4525
+ psubw m2, m5526
+ psubw m3, m6527
+ pabsw m1, m1528
+ pabsw m2, m2529
+ pabsw m3, m3530
+ paddw m1, m2531
+ paddw m7, m3, m1532
+533
+ movu m1, [r2 + 2 * r3]534
+ movu m2, [r2 + r5]535
+ movu ym3, [r2 + 2 * r3 + mmsize]536
+ vinserti32x8 m3, [r2 + r5 + mmsize], 1537
+ movu m4, [r0 + 2 * r1]538
+ movu m5, [r0 + r4]539
+ movu ym6, [r0 + 2 * r1 + mmsize]540
+ vinserti32x8 m6, [r0 + r4 + mmsize], 1541
+ psubw m1, m4542
+ psubw m2, m5543
+ psubw m3, m6544
+ pabsw m1, m1545
+ pabsw m2, m2546
+ pabsw m3, m3547
+ paddw m1, m2548
+ paddw m1, m3549
+550
+ pmaddwd m7, m8551
+ paddd m0, m7552
+ pmaddwd m1, m8553
+ paddd m0, m1554
+ lea r0, [r0 + 4 * r1]555
+ lea r2, [r2 + 4 * r3]556
+557
+ dec r6d558
+ jg .loop559
+560
+ PROCESS_SAD_AVX512_END561
+ RET562
+%endif563
+564
;=============================================================================565
; SAD x3/x4566
;=============================================================================567
568
SAD_X 4, 64, 48569
SAD_X 4, 64, 64570
571
+;============================572
+; SAD x3/x4 avx512 code start573
+;============================574
+575
+%macro PROCESS_SAD_X4_16x4_AVX512 0576
+ movu ym8, [r0]577
+ vinserti64x4 m8, [r0 + 2 * FENC_STRIDE], 1578
+ movu ym4, [r1]579
+ vinserti64x4 m4, [r1 + r5], 1580
+ movu ym5, [r2]581
+ vinserti64x4 m5, [r2 + r5], 1582
+ movu ym6, [r3]583
+ vinserti64x4 m6, [r3 + r5], 1584
+ movu ym7, [r4]585
+ vinserti64x4 m7, [r4 + r5], 1586
+587
+ 588
+ psubw m4, m8589
+ psubw m5, m8590
+ psubw m6, m8591
+ psubw m7, m8592
+ pabsw m4, m4593
+ pabsw m5, m5594
+ pabsw m6, m6595
+ pabsw m7, m7596
+597
+ pmaddwd m4, m9598
+ paddd m0, m4599
+ pmaddwd m5, m9600
+ paddd m1, m5601
+ pmaddwd m6, m9602
+ paddd m2, m6603
+ pmaddwd m7, m9604
+ paddd m3, m7605
+606
+ movu ym8, [r0 + 4 * FENC_STRIDE]607
+ vinserti64x4 m8, [r0 + 6 * FENC_STRIDE], 1608
+ movu ym4, [r1 + 2 * r5]609
+ vinserti64x4 m4, [r1 + r7], 1610
+ movu ym5, [r2 + 2 * r5]611
+ vinserti64x4 m5, [r2 + r7], 1612
+ movu ym6, [r3 + 2 * r5]613
+ vinserti64x4 m6, [r3 + r7], 1614
+ movu ym7, [r4 + 2 * r5]615
+ vinserti64x4 m7, [r4 + r7], 1616
+617
+ psubw m4, m8618
+ psubw m5, m8619
+ psubw m6, m8620
+ psubw m7, m8621
+ pabsw m4, m4622
+ pabsw m5, m5623
+ pabsw m6, m6624
+ pabsw m7, m7625
+626
+ pmaddwd m4, m9627
+ paddd m0, m4628
+ pmaddwd m5, m9629
+ paddd m1, m5630
+ pmaddwd m6, m9631
+ paddd m2, m6632
+ pmaddwd m7, m9633
+ paddd m3, m7634
+%endmacro635
+636
+%macro PROCESS_SAD_X4_32x4_AVX512 0637
+ movu m8, [r0]638
+ movu m4, [r1]639
+ movu m5, [r2]640
+ movu m6, [r3]641
+ movu m7, [r4]642
+643
+ 644
+ psubw m4, m8645
+ psubw m5, m8646
+ psubw m6, m8647
+ psubw m7, m8648
+ pabsw m4, m4649
+ pabsw m5, m5650
+ pabsw m6, m6651
+ pabsw m7, m7652
+653
+ pmaddwd m4, m9654
+ paddd m0, m4655
+ pmaddwd m5, m9656
+ paddd m1, m5657
+ pmaddwd m6, m9658
+ paddd m2, m6659
+ pmaddwd m7, m9660
+ paddd m3, m7661
+662
+663
+ movu m8, [r0 + 2 * FENC_STRIDE]664
+ movu m4, [r1 + r5]665
+ movu m5, [r2 + r5]666
+ movu m6, [r3 + r5]667
+ movu m7, [r4 + r5]668
+669
+ 670
+ psubw m4, m8671
+ psubw m5, m8672
+ psubw m6, m8673
+ psubw m7, m8674
+ pabsw m4, m4675
+ pabsw m5, m5676
+ pabsw m6, m6677
+ pabsw m7, m7678
+679
+ pmaddwd m4, m9680
+ paddd m0, m4681
+ pmaddwd m5, m9682
+ paddd m1, m5683
+ pmaddwd m6, m9684
+ paddd m2, m6685
+ pmaddwd m7, m9686
+ paddd m3, m7687
+688
+ movu m8, [r0 + 4 * FENC_STRIDE]689
+ movu m4, [r1 + 2 * r5]690
+ movu m5, [r2 + 2 * r5]691
+ movu m6, [r3 + 2 * r5]692
+ movu m7, [r4 + 2 * r5]693
+694
+ 695
+ psubw m4, m8696
+ psubw m5, m8697
+ psubw m6, m8698
+ psubw m7, m8699
+ pabsw m4, m4700
+ pabsw m5, m5701
+ pabsw m6, m6702
+ pabsw m7, m7703
+704
+ pmaddwd m4, m9705
+ paddd m0, m4706
+ pmaddwd m5, m9707
+ paddd m1, m5708
+ pmaddwd m6, m9709
+ paddd m2, m6710
+ pmaddwd m7, m9711
+ paddd m3, m7712
+713
+ movu m8, [r0 + 6 * FENC_STRIDE]714
+ movu m4, [r1 + r7]715
+ movu m5, [r2 + r7]716
+ movu m6, [r3 + r7]717
+ movu m7, [r4 + r7]718
+719
+ 720
+ psubw m4, m8721
+ psubw m5, m8722
+ psubw m6, m8723
+ psubw m7, m8724
+ pabsw m4, m4725
+ pabsw m5, m5726
+ pabsw m6, m6727
+ pabsw m7, m7728
+729
+ pmaddwd m4, m9730
+ paddd m0, m4731
+ pmaddwd m5, m9732
+ paddd m1, m5733
+ pmaddwd m6, m9734
+ paddd m2, m6735
+ pmaddwd m7, m9736
+ paddd m3, m7737
+%endmacro738
+739
+%macro PROCESS_SAD_X4_64x4_AVX512 0740
+ movu m8, [r0]741
+ movu m10, [r0 + mmsize]742
+ movu m4, [r1]743
+ movu m11, [r1 + mmsize]744
+ movu m5, [r2]745
+ movu m12, [r2 + mmsize]746
+ movu m6, [r3]747
+ movu m13, [r3 + mmsize]748
+ movu m7, [r4]749
+ movu m14, [r4 + mmsize]750
+751
+ psubw m4, m8752
+ psubw m5, m8753
+ psubw m6, m8754
+ psubw m7, m8755
+ psubw m11, m10756
+ psubw m12, m10757
+ psubw m13, m10758
+ psubw m14, m10759
+ pabsw m4, m4760
+ pabsw m5, m5761
+ pabsw m6, m6762
+ pabsw m7, m7763
+ pabsw m11, m11764
+ pabsw m12, m12765
+ pabsw m13, m13766
+ pabsw m14, m14767
+ paddw m4, m11768
+ paddw m5, m12769
+ paddw m6, m13770
+ paddw m7, m14771
+772
+ pmaddwd m4, m9773
+ paddd m0, m4774
+ pmaddwd m5, m9775
+ paddd m1, m5776
+ pmaddwd m6, m9777
+ paddd m2, m6778
+ pmaddwd m7, m9779
+ paddd m3, m7780
+781
+782
+ movu m8, [r0 + 2 * FENC_STRIDE]783
+ movu m10, [r0 + 2 * FENC_STRIDE + mmsize]784
+ movu m4, [r1 + r5]785
+ movu m11, [r1 + r5 + mmsize]786
+ movu m5, [r2 + r5]787
+ movu m12, [r2 + r5 + mmsize]788
+ movu m6, [r3 + r5]789
+ movu m13, [r3 + r5 + mmsize]790
+ movu m7, [r4 + r5]791
+ movu m14, [r4 + r5 + mmsize]792
+793
+ psubw m4, m8794
+ psubw m5, m8795
+ psubw m6, m8796
+ psubw m7, m8797
+ psubw m11, m10798
+ psubw m12, m10799
+ psubw m13, m10800
+ psubw m14, m10801
+ pabsw m4, m4802
+ pabsw m5, m5803
+ pabsw m6, m6804
+ pabsw m7, m7805
+ pabsw m11, m11806
+ pabsw m12, m12807
+ pabsw m13, m13808
+ pabsw m14, m14809
+ paddw m4, m11810
+ paddw m5, m12811
+ paddw m6, m13812
+ paddw m7, m14813
+814
+ pmaddwd m4, m9815
+ paddd m0, m4816
+ pmaddwd m5, m9817
+ paddd m1, m5818
+ pmaddwd m6, m9819
+ paddd m2, m6820
+ pmaddwd m7, m9821
+ paddd m3, m7822
+823
+ movu m8, [r0 + 4 * FENC_STRIDE]824
+ movu m10, [r0 + 4 * FENC_STRIDE + mmsize]825
+ movu m4, [r1 + 2 * r5]826
+ movu m11, [r1 + 2 * r5 + mmsize]827
+ movu m5, [r2 + 2 * r5]828
+ movu m12, [r2 + 2 * r5 + mmsize]829
+ movu m6, [r3 + 2 * r5]830
+ movu m13, [r3 + 2 * r5 + mmsize]831
+ movu m7, [r4 + 2 * r5]832
+ movu m14, [r4 + 2 * r5 + mmsize]833
+834
+ psubw m4, m8835
+ psubw m5, m8836
+ psubw m6, m8837
+ psubw m7, m8838
+ psubw m11, m10839
+ psubw m12, m10840
+ psubw m13, m10841
+ psubw m14, m10842
+ pabsw m4, m4843
+ pabsw m5, m5844
+ pabsw m6, m6845
+ pabsw m7, m7846
+ pabsw m11, m11847
+ pabsw m12, m12848
+ pabsw m13, m13849
+ pabsw m14, m14850
+ paddw m4, m11851
+ paddw m5, m12852
+ paddw m6, m13853
+ paddw m7, m14854
+855
+ pmaddwd m4, m9856
+ paddd m0, m4857
+ pmaddwd m5, m9858
+ paddd m1, m5859
+ pmaddwd m6, m9860
+ paddd m2, m6861
+ pmaddwd m7, m9862
+ paddd m3, m7863
+864
+ movu m8, [r0 + 6 * FENC_STRIDE]865
+ movu m10, [r0 + 6 * FENC_STRIDE + mmsize]866
+ movu m4, [r1 + r7]867
+ movu m11, [r1 + r7 + mmsize]868
+ movu m5, [r2 + r7]869
+ movu m12, [r2 + r7 + mmsize]870
+ movu m6, [r3 + r7]871
+ movu m13, [r3 + r7 + mmsize]872
+ movu m7, [r4 + r7]873
+ movu m14, [r4 + r7 + mmsize]874
+875
+ psubw m4, m8876
+ psubw m5, m8877
+ psubw m6, m8878
+ psubw m7, m8879
+ psubw m11, m10880
+ psubw m12, m10881
+ psubw m13, m10882
+ psubw m14, m10883
+ pabsw m4, m4884
+ pabsw m5, m5885
+ pabsw m6, m6886
+ pabsw m7, m7887
+ pabsw m11, m11888
+ pabsw m12, m12889
+ pabsw m13, m13890
+ pabsw m14, m14891
+ paddw m4, m11892
+ paddw m5, m12893
+ paddw m6, m13894
+ paddw m7, m14895
+896
+ pmaddwd m4, m9897
+ paddd m0, m4898
+ pmaddwd m5, m9899
+ paddd m1, m5900
+ pmaddwd m6, m9901
+ paddd m2, m6902
+ pmaddwd m7, m9903
+ paddd m3, m7904
+%endmacro905
+906
+%macro PROCESS_SAD_X4_END_AVX512 0907
+ vextracti32x8 ym4, m0, 1908
+ vextracti32x8 ym5, m1, 1909
+ vextracti32x8 ym6, m2, 1910
+ vextracti32x8 ym7, m3, 1911
+912
+ paddd ym0, ym4913
+ paddd ym1, ym5914
+ paddd ym2, ym6915
+ paddd ym3, ym7916
+917
+ vextracti64x2 xm4, m0, 1918
+ vextracti64x2 xm5, m1, 1919
+ vextracti64x2 xm6, m2, 1920
+ vextracti64x2 xm7, m3, 1921
+922
+ paddd xm0, xm4923
+ paddd xm1, xm5924
+ paddd xm2, xm6925
+ paddd xm3, xm7926
+927
+ pshufd xm4, xm0, 00001110b928
+ pshufd xm5, xm1, 00001110b929
+ pshufd xm6, xm2, 00001110b930
+ pshufd xm7, xm3, 00001110b931
+932
+ paddd xm0, xm4933
+ paddd xm1, xm5934
+ paddd xm2, xm6935
+ paddd xm3, xm7936
+937
+ pshufd xm4, xm0, 00000001b938
+ pshufd xm5, xm1, 00000001b939
+ pshufd xm6, xm2, 00000001b940
+ pshufd xm7, xm3, 00000001b941
+942
+ paddd xm0, xm4943
+ paddd xm1, xm5944
+ paddd xm2, xm6945
+ paddd xm3, xm7946
+947
+ mov r0, r6mp948
+ movd [r0 + 0], xm0949
+ movd [r0 + 4], xm1950
+ movd [r0 + 8], xm2951
+ movd [r0 + 12], xm3952
+%endmacro953
+954
+955
+%macro PROCESS_SAD_X3_16x4_AVX512 0956
+ movu ym6, [r0]957
+ vinserti64x4 m6, [r0 + 2 * FENC_STRIDE], 1958
+ movu ym3, [r1]959
+ vinserti64x4 m3, [r1 + r4], 1960
+ movu ym4, [r2]961
+ vinserti64x4 m4, [r2 + r4], 1962
+ movu ym5, [r3]963
+ vinserti64x4 m5, [r3 + r4], 1964
+965
+ psubw m3, m6966
+ psubw m4, m6967
+ psubw m5, m6968
+ pabsw m3, m3969
+ pabsw m4, m4970
+ pabsw m5, m5971
+972
+ pmaddwd m3, m7973
+ paddd m0, m3974
+ pmaddwd m4, m7975
+ paddd m1, m4976
+ pmaddwd m5, m7977
+ paddd m2, m5978
+979
+ movu ym6, [r0 + 4 * FENC_STRIDE]980
+ vinserti64x4 m6, [r0 + 6 * FENC_STRIDE], 1981
+ movu ym3, [r1 + 2 * r4]982
+ vinserti64x4 m3, [r1 + r6], 1983
+ movu ym4, [r2 + 2 * r4]984
+ vinserti64x4 m4, [r2 + r6], 1985
+ movu ym5, [r3 + 2 * r4]986
+ vinserti64x4 m5, [r3 + r6], 1987
+988
+ psubw m3, m6989
+ psubw m4, m6990
+ psubw m5, m6991
+ pabsw m3, m3992
+ pabsw m4, m4993
+ pabsw m5, m5994
+995
+ pmaddwd m3, m7996
+ paddd m0, m3997
+ pmaddwd m4, m7998
+ paddd m1, m4999
+ pmaddwd m5, m71000
+ paddd m2, m51001
+%endmacro1002
+1003
+1004
+%macro PROCESS_SAD_X3_32x4_AVX512 01005
+ movu m6, [r0]1006
+ movu m3, [r1]1007
+ movu m4, [r2]1008
+ movu m5, [r3]1009
+1010
+ 1011
+ psubw m3, m61012
+ psubw m4, m61013
+ psubw m5, m61014
+ pabsw m3, m31015
+ pabsw m4, m41016
+ pabsw m5, m51017
+1018
+ pmaddwd m3, m71019
+ paddd m0, m31020
+ pmaddwd m4, m71021
+ paddd m1, m41022
+ pmaddwd m5, m71023
+ paddd m2, m51024
+1025
+ movu m6, [r0 + 2 * FENC_STRIDE]1026
+ movu m3, [r1 + r4]1027
+ movu m4, [r2 + r4]1028
+ movu m5, [r3 + r4]1029
+ 1030
+ psubw m3, m61031
+ psubw m4, m61032
+ psubw m5, m61033
+ pabsw m3, m31034
+ pabsw m4, m41035
+ pabsw m5, m51036
+1037
+ pmaddwd m3, m71038
+ paddd m0, m31039
+ pmaddwd m4, m71040
+ paddd m1, m41041
+ pmaddwd m5, m71042
+ paddd m2, m51043
+1044
+ movu m6, [r0 + 4 * FENC_STRIDE]1045
+ movu m3, [r1 + 2 * r4]1046
+ movu m4, [r2 + 2 * r4]1047
+ movu m5, [r3 + 2 * r4]1048
+ 1049
+ psubw m3, m61050
+ psubw m4, m61051
+ psubw m5, m61052
+ pabsw m3, m31053
+ pabsw m4, m41054
+ pabsw m5, m51055
+1056
+ pmaddwd m3, m71057
+ paddd m0, m31058
+ pmaddwd m4, m71059
+ paddd m1, m41060
+ pmaddwd m5, m71061
+ paddd m2, m51062
+1063
+ movu m6, [r0 + 6 * FENC_STRIDE]1064
+ movu m3, [r1 + r6]1065
+ movu m4, [r2 + r6]1066
+ movu m5, [r3 + r6]1067
+ 1068
+ psubw m3, m61069
+ psubw m4, m61070
+ psubw m5, m61071
+ pabsw m3, m31072
+ pabsw m4, m41073
+ pabsw m5, m51074
+1075
+ pmaddwd m3, m71076
+ paddd m0, m31077
+ pmaddwd m4, m71078
+ paddd m1, m41079
+ pmaddwd m5, m71080
+ paddd m2, m51081
+%endmacro1082
+1083
+%macro PROCESS_SAD_X3_64x4_AVX512 01084
+ movu m6, [r0]1085
+ movu m8, [r0 + mmsize]1086
+ movu m3, [r1]1087
+ movu m9, [r1 + mmsize]1088
+ movu m4, [r2]1089
+ movu m10, [r2 + mmsize]1090
+ movu m5, [r3]1091
+ movu m11, [r3 + mmsize]1092
+1093
+ psubw m3, m61094
+ psubw m9, m81095
+ psubw m4, m61096
+ psubw m10, m81097
+ psubw m5, m61098
+ psubw m11, m81099
+ pabsw m3, m31100
+ pabsw m4, m41101
+ pabsw m5, m51102
+ pabsw m9, m91103
+ pabsw m10, m101104
+ pabsw m11, m111105
+ paddw m3, m91106
+ paddw m4, m101107
+ paddw m5, m111108
+1109
+ pmaddwd m3, m71110
+ paddd m0, m31111
+ pmaddwd m4, m71112
+ paddd m1, m41113
+ pmaddwd m5, m71114
+ paddd m2, m51115
+1116
+ movu m6, [r0 + 2 * FENC_STRIDE]1117
+ movu m8, [r0 + 2 * FENC_STRIDE + mmsize]1118
+ movu m3, [r1 + r4]1119
+ movu m9, [r1 + r4 + mmsize]1120
+ movu m4, [r2 + r4]1121
+ movu m10, [r2 + r4 + mmsize]1122
+ movu m5, [r3 + r4]1123
+ movu m11, [r3 + r4 + mmsize]1124
+1125
+ psubw m3, m61126
+ psubw m9, m81127
+ psubw m4, m61128
+ psubw m10, m81129
+ psubw m5, m61130
+ psubw m11, m81131
+ pabsw m3, m31132
+ pabsw m4, m41133
+ pabsw m5, m51134
+ pabsw m9, m91135
+ pabsw m10, m101136
+ pabsw m11, m111137
+ paddw m3, m91138
+ paddw m4, m101139
+ paddw m5, m111140
+1141
+ pmaddwd m3, m71142
+ paddd m0, m31143
+ pmaddwd m4, m71144
+ paddd m1, m41145
+ pmaddwd m5, m71146
+ paddd m2, m51147
+1148
+ movu m6, [r0 + 4 * FENC_STRIDE]1149
+ movu m8, [r0 + 4 * FENC_STRIDE + mmsize]1150
+ movu m3, [r1 + 2 * r4]1151
+ movu m9, [r1 + 2 * r4 + mmsize]1152
+ movu m4, [r2 + 2 * r4]1153
+ movu m10, [r2 + 2 * r4 + mmsize]1154
+ movu m5, [r3 + 2 * r4]1155
+ movu m11, [r3 + 2 * r4 + mmsize]1156
+1157
+ psubw m3, m61158
+ psubw m9, m81159
+ psubw m4, m61160
+ psubw m10, m81161
+ psubw m5, m61162
+ psubw m11, m81163
+ pabsw m3, m31164
+ pabsw m4, m41165
+ pabsw m5, m51166
+ pabsw m9, m91167
+ pabsw m10, m101168
+ pabsw m11, m111169
+ paddw m3, m91170
+ paddw m4, m101171
+ paddw m5, m111172
+1173
+ pmaddwd m3, m71174
+ paddd m0, m31175
+ pmaddwd m4, m71176
+ paddd m1, m41177
+ pmaddwd m5, m71178
+ paddd m2, m51179
+1180
+ movu m6, [r0 + 6 * FENC_STRIDE]1181
+ movu m8, [r0 + 6 * FENC_STRIDE + mmsize]1182
+ movu m3, [r1 + r6]1183
+ movu m9, [r1 + r6 + mmsize]1184
+ movu m4, [r2 + r6]1185
+ movu m10, [r2 + r6 + mmsize]1186
+ movu m5, [r3 + r6]1187
+ movu m11, [r3 + r6 + mmsize]1188
+1189
+ psubw m3, m61190
+ psubw m9, m81191
+ psubw m4, m61192
+ psubw m10, m81193
+ psubw m5, m61194
+ psubw m11, m81195
+ pabsw m3, m31196
+ pabsw m4, m41197
+ pabsw m5, m51198
+ pabsw m9, m91199
+ pabsw m10, m101200
+ pabsw m11, m111201
+ paddw m3, m91202
+ paddw m4, m101203
+ paddw m5, m111204
+1205
+ pmaddwd m3, m71206
+ paddd m0, m31207
+ pmaddwd m4, m71208
+ paddd m1, m41209
+ pmaddwd m5, m71210
+ paddd m2, m51211
+%endmacro1212
+1213
+%macro PROCESS_SAD_X3_END_AVX512 01214
+ vextracti32x8 ym3, m0, 11215
+ vextracti32x8 ym4, m1, 11216
+ vextracti32x8 ym5, m2, 11217
+ 1218
+ paddd ym0, ym31219
+ paddd ym1, ym41220
+ paddd ym2, ym51221
+1222
+ vextracti64x2 xm3, m0, 11223
+ vextracti64x2 xm4, m1, 11224
+ vextracti64x2 xm5, m2, 11225
+1226
+ paddd xm0, xm31227
+ paddd xm1, xm41228
+ paddd xm2, xm51229
+1230
+ pshufd xm3, xm0, 00001110b1231
+ pshufd xm4, xm1, 00001110b1232
+ pshufd xm5, xm2, 00001110b1233
+1234
+ paddd xm0, xm31235
+ paddd xm1, xm41236
+ paddd xm2, xm51237
+1238
+ pshufd xm3, xm0, 00000001b1239
+ pshufd xm4, xm1, 00000001b1240
+ pshufd xm5, xm2, 00000001b1241
+1242
+ paddd xm0, xm31243
+ paddd xm1, xm41244
+ paddd xm2, xm51245
+1246
+ %if UNIX641247
+ movd [r5 + 0], xm01248
+ movd [r5 + 4], xm11249
+ movd [r5 + 8], xm21250
+ %else1251
+ mov r0, r5mp1252
+ movd [r0 + 0], xm01253
+ movd [r0 + 4], xm11254
+ movd [r0 + 8], xm21255
+%endif1256
+%endmacro1257
+1258
+1259
+;------------------------------------------------------------------------------------------------------------------------------------------1260
+; void pixel_sad_x3_16x%1( const pixel* pix1, const pixel* pix2, const pixel* pix3, const pixel* pix4, intptr_t frefstride, int32_t* res )1261
+;------------------------------------------------------------------------------------------------------------------------------------------1262
+%if ARCH_X86_641263
+INIT_ZMM avx5121264
+cglobal pixel_sad_x3_16x8, 6,7,81265
+ pxor m0, m01266
+ pxor m1, m11267
+ pxor m2, m21268
+1269
+ vbroadcasti32x8 m7, [pw_1]1270
+1271
+ add r4d, r4d1272
+ lea r6d, [r4 * 3]1273
+1274
+ PROCESS_SAD_X3_16x4_AVX5121275
+ add r0, FENC_STRIDE * 81276
+ lea r1, [r1 + r4 * 4]1277
+ lea r2, [r2 + r4 * 4]1278
+ lea r3, [r3 + r4 * 4]1279
+ PROCESS_SAD_X3_16x4_AVX5121280
+ PROCESS_SAD_X3_END_AVX5121281
+ RET1282
+1283
+INIT_ZMM avx5121284
+cglobal pixel_sad_x3_16x12, 6,7,81285
+ pxor m0, m01286
+ pxor m1, m11287
+ pxor m2, m21288
+1289
+ vbroadcasti32x8 m7, [pw_1]1290
+1291
+ add r4d, r4d1292
+ lea r6d, [r4 * 3]1293
+ %rep 21294
+ PROCESS_SAD_X3_16x4_AVX5121295
+ add r0, FENC_STRIDE * 81296
+ lea r1, [r1 + r4 * 4]1297
+ lea r2, [r2 + r4 * 4]1298
+ lea r3, [r3 + r4 * 4]1299
+ %endrep1300
+ PROCESS_SAD_X3_16x4_AVX5121301
+ PROCESS_SAD_X3_END_AVX5121302
+ RET1303
+1304
+INIT_ZMM avx5121305
+cglobal pixel_sad_x3_16x16, 6,7,81306
+ pxor m0, m01307
+ pxor m1, m11308
+ pxor m2, m21309
+1310
+ vbroadcasti32x8 m7, [pw_1]1311
+1312
+ add r4d, r4d1313
+ lea r6d, [r4 * 3]1314
+1315
+ %rep 31316
+ PROCESS_SAD_X3_16x4_AVX5121317
+ add r0, FENC_STRIDE * 81318
+ lea r1, [r1 + r4 * 4]1319
+ lea r2, [r2 + r4 * 4]1320
+ lea r3, [r3 + r4 * 4]1321
+ %endrep1322
+ PROCESS_SAD_X3_16x4_AVX5121323
+ PROCESS_SAD_X3_END_AVX5121324
+ RET1325
+1326
+INIT_ZMM avx5121327
+cglobal pixel_sad_x3_16x32, 6,7,81328
+ pxor m0, m01329
+ pxor m1, m11330
+ pxor m2, m21331
+1332
+ vbroadcasti32x8 m7, [pw_1]1333
+1334
+ add r4d, r4d1335
+ lea r6d, [r4 * 3]1336
+1337
+ %rep 71338
+ PROCESS_SAD_X3_16x4_AVX5121339
+ add r0, FENC_STRIDE * 81340
+ lea r1, [r1 + r4 * 4]1341
+ lea r2, [r2 + r4 * 4]1342
+ lea r3, [r3 + r4 * 4]1343
+ %endrep1344
+ PROCESS_SAD_X3_16x4_AVX5121345
+ PROCESS_SAD_X3_END_AVX5121346
+ RET1347
+1348
+INIT_ZMM avx5121349
+cglobal pixel_sad_x3_16x64, 6,7,81350
+ pxor m0, m01351
+ pxor m1, m11352
+ pxor m2, m21353
+1354
+ vbroadcasti32x8 m7, [pw_1]1355
+1356
+ add r4d, r4d1357
+ lea r6d, [r4 * 3]1358
+1359
+ %rep 151360
+ PROCESS_SAD_X3_16x4_AVX5121361
+ add r0, FENC_STRIDE * 81362
+ lea r1, [r1 + r4 * 4]1363
+ lea r2, [r2 + r4 * 4]1364
+ lea r3, [r3 + r4 * 4]1365
+ %endrep1366
+ PROCESS_SAD_X3_16x4_AVX5121367
+ PROCESS_SAD_X3_END_AVX5121368
+ RET1369
+%endif1370
+1371
+;------------------------------------------------------------------------------------------------------------------------------------------1372
+; void pixel_sad_x3_32x%1( const pixel* pix1, const pixel* pix2, const pixel* pix3, const pixel* pix4, intptr_t frefstride, int32_t* res )1373
+;------------------------------------------------------------------------------------------------------------------------------------------1374
+%if ARCH_X86_641375
+INIT_ZMM avx5121376
+cglobal pixel_sad_x3_32x8, 6,7,81377
+ pxor m0, m01378
+ pxor m1, m11379
+ pxor m2, m21380
+1381
+ vbroadcasti32x8 m7, [pw_1]1382
+1383
+ add r4d, r4d1384
+ lea r6d, [r4 * 3]1385
+1386
+ PROCESS_SAD_X3_32x4_AVX5121387
+ add r0, FENC_STRIDE * 81388
+ lea r1, [r1 + r4 * 4]1389
+ lea r2, [r2 + r4 * 4]1390
+ lea r3, [r3 + r4 * 4]1391
+ PROCESS_SAD_X3_32x4_AVX5121392
+ PROCESS_SAD_X3_END_AVX5121393
+ RET1394
+1395
+1396
+INIT_ZMM avx5121397
+cglobal pixel_sad_x3_32x16, 6,7,81398
+ pxor m0, m01399
+ pxor m1, m11400
+ pxor m2, m21401
+1402
+ vbroadcasti32x8 m7, [pw_1]1403
+1404
+ add r4d, r4d1405
+ lea r6d, [r4 * 3]1406
+1407
+ PROCESS_SAD_X3_32x4_AVX5121408
+ add r0, FENC_STRIDE * 81409
+ lea r1, [r1 + r4 * 4]1410
+ lea r2, [r2 + r4 * 4]1411
+ lea r3, [r3 + r4 * 4]1412
+ PROCESS_SAD_X3_32x4_AVX5121413
+ add r0, FENC_STRIDE * 81414
+ lea r1, [r1 + r4 * 4]1415
+ lea r2, [r2 + r4 * 4]1416
+ lea r3, [r3 + r4 * 4]1417
+ PROCESS_SAD_X3_32x4_AVX5121418
+ add r0, FENC_STRIDE * 81419
+ lea r1, [r1 + r4 * 4]1420
+ lea r2, [r2 + r4 * 4]1421
+ lea r3, [r3 + r4 * 4]1422
+ PROCESS_SAD_X3_32x4_AVX5121423
+ PROCESS_SAD_X3_END_AVX5121424
+ RET1425
+1426
+INIT_ZMM avx5121427
+cglobal pixel_sad_x3_32x24, 6,7,81428
+ pxor m0, m01429
+ pxor m1, m11430
+ pxor m2, m21431
+1432
+ vbroadcasti32x8 m7, [pw_1]1433
+1434
+ add r4d, r4d1435
+ lea r6d, [r4 * 3]1436
+1437
+ PROCESS_SAD_X3_32x4_AVX5121438
+ add r0, FENC_STRIDE * 81439
+ lea r1, [r1 + r4 * 4]1440
+ lea r2, [r2 + r4 * 4]1441
+ lea r3, [r3 + r4 * 4]1442
+ PROCESS_SAD_X3_32x4_AVX5121443
+ add r0, FENC_STRIDE * 81444
+ lea r1, [r1 + r4 * 4]1445
+ lea r2, [r2 + r4 * 4]1446
+ lea r3, [r3 + r4 * 4]1447
+ PROCESS_SAD_X3_32x4_AVX5121448
+ add r0, FENC_STRIDE * 81449
+ lea r1, [r1 + r4 * 4]1450
+ lea r2, [r2 + r4 * 4]1451
+ lea r3, [r3 + r4 * 4]1452
+ PROCESS_SAD_X3_32x4_AVX5121453
+ add r0, FENC_STRIDE * 81454
+ lea r1, [r1 + r4 * 4]1455
+ lea r2, [r2 + r4 * 4]1456
+ lea r3, [r3 + r4 * 4]1457
+ PROCESS_SAD_X3_32x4_AVX5121458
+ add r0, FENC_STRIDE * 81459
+ lea r1, [r1 + r4 * 4]1460
+ lea r2, [r2 + r4 * 4]1461
+ lea r3, [r3 + r4 * 4]1462
+ PROCESS_SAD_X3_32x4_AVX5121463
+ PROCESS_SAD_X3_END_AVX5121464
+ RET1465
+1466
+1467
+INIT_ZMM avx5121468
+cglobal pixel_sad_x3_32x32, 6,7,81469
+ pxor m0, m01470
+ pxor m1, m11471
+ pxor m2, m21472
+1473
+ vbroadcasti32x8 m7, [pw_1]1474
+1475
+ add r4d, r4d1476
+ lea r6d, [r4 * 3]1477
+1478
+ PROCESS_SAD_X3_32x4_AVX5121479
+ add r0, FENC_STRIDE * 81480
+ lea r1, [r1 + r4 * 4]1481
+ lea r2, [r2 + r4 * 4]1482
+ lea r3, [r3 + r4 * 4]1483
+ PROCESS_SAD_X3_32x4_AVX5121484
+ add r0, FENC_STRIDE * 81485
+ lea r1, [r1 + r4 * 4]1486
+ lea r2, [r2 + r4 * 4]1487
+ lea r3, [r3 + r4 * 4]1488
+ PROCESS_SAD_X3_32x4_AVX5121489
+ add r0, FENC_STRIDE * 81490
+ lea r1, [r1 + r4 * 4]1491
+ lea r2, [r2 + r4 * 4]1492
+ lea r3, [r3 + r4 * 4]1493
+ PROCESS_SAD_X3_32x4_AVX5121494
+ add r0, FENC_STRIDE * 81495
+ lea r1, [r1 + r4 * 4]1496
+ lea r2, [r2 + r4 * 4]1497
+ lea r3, [r3 + r4 * 4]1498
+ PROCESS_SAD_X3_32x4_AVX5121499
+ add r0, FENC_STRIDE * 81500
+ lea r1, [r1 + r4 * 4]1501
+ lea r2, [r2 + r4 * 4]1502
+ lea r3, [r3 + r4 * 4]1503
+ PROCESS_SAD_X3_32x4_AVX5121504
+ add r0, FENC_STRIDE * 81505
+ lea r1, [r1 + r4 * 4]1506
+ lea r2, [r2 + r4 * 4]1507
+ lea r3, [r3 + r4 * 4]1508
+ PROCESS_SAD_X3_32x4_AVX5121509
+ add r0, FENC_STRIDE * 81510
+ lea r1, [r1 + r4 * 4]1511
+ lea r2, [r2 + r4 * 4]1512
+ lea r3, [r3 + r4 * 4]1513
+ PROCESS_SAD_X3_32x4_AVX5121514
+ PROCESS_SAD_X3_END_AVX5121515
+ RET1516
+1517
+INIT_ZMM avx5121518
+cglobal pixel_sad_x3_32x64, 6,7,81519
+ pxor m0, m01520
+ pxor m1, m11521
+ pxor m2, m21522
+1523
+ vbroadcasti32x8 m7, [pw_1]1524
+1525
+ add r4d, r4d1526
+ lea r6d, [r4 * 3]1527
+1528
+ PROCESS_SAD_X3_32x4_AVX5121529
+ add r0, FENC_STRIDE * 81530
+ lea r1, [r1 + r4 * 4]1531
+ lea r2, [r2 + r4 * 4]1532
+ lea r3, [r3 + r4 * 4]1533
+ PROCESS_SAD_X3_32x4_AVX5121534
+ add r0, FENC_STRIDE * 81535
+ lea r1, [r1 + r4 * 4]1536
+ lea r2, [r2 + r4 * 4]1537
+ lea r3, [r3 + r4 * 4]1538
+ PROCESS_SAD_X3_32x4_AVX5121539
+ add r0, FENC_STRIDE * 81540
+ lea r1, [r1 + r4 * 4]1541
+ lea r2, [r2 + r4 * 4]1542
+ lea r3, [r3 + r4 * 4]1543
+ PROCESS_SAD_X3_32x4_AVX5121544
+ add r0, FENC_STRIDE * 81545
+ lea r1, [r1 + r4 * 4]1546
+ lea r2, [r2 + r4 * 4]1547
+ lea r3, [r3 + r4 * 4]1548
+ PROCESS_SAD_X3_32x4_AVX5121549
+ add r0, FENC_STRIDE * 81550
+ lea r1, [r1 + r4 * 4]1551
+ lea r2, [r2 + r4 * 4]1552
+ lea r3, [r3 + r4 * 4]1553
+ PROCESS_SAD_X3_32x4_AVX5121554
+ add r0, FENC_STRIDE * 81555
+ lea r1, [r1 + r4 * 4]1556
+ lea r2, [r2 + r4 * 4]1557
+ lea r3, [r3 + r4 * 4]1558
+ PROCESS_SAD_X3_32x4_AVX5121559
+ add r0, FENC_STRIDE * 81560
+ lea r1, [r1 + r4 * 4]1561
+ lea r2, [r2 + r4 * 4]1562
+ lea r3, [r3 + r4 * 4]1563
+ PROCESS_SAD_X3_32x4_AVX5121564
+ add r0, FENC_STRIDE * 81565
+ lea r1, [r1 + r4 * 4]1566
+ lea r2, [r2 + r4 * 4]1567
+ lea r3, [r3 + r4 * 4]1568
+ PROCESS_SAD_X3_32x4_AVX5121569
+ add r0, FENC_STRIDE * 81570
+ lea r1, [r1 + r4 * 4]1571
+ lea r2, [r2 + r4 * 4]1572
+ lea r3, [r3 + r4 * 4]1573
+ PROCESS_SAD_X3_32x4_AVX5121574
+ add r0, FENC_STRIDE * 81575
+ lea r1, [r1 + r4 * 4]1576
+ lea r2, [r2 + r4 * 4]1577
+ lea r3, [r3 + r4 * 4]1578
+ PROCESS_SAD_X3_32x4_AVX5121579
+ add r0, FENC_STRIDE * 81580
+ lea r1, [r1 + r4 * 4]1581
+ lea r2, [r2 + r4 * 4]1582
+ lea r3, [r3 + r4 * 4]1583
+ PROCESS_SAD_X3_32x4_AVX5121584
+ add r0, FENC_STRIDE * 81585
+ lea r1, [r1 + r4 * 4]1586
+ lea r2, [r2 + r4 * 4]1587
+ lea r3, [r3 + r4 * 4]1588
+ PROCESS_SAD_X3_32x4_AVX5121589
+ add r0, FENC_STRIDE * 81590
+ lea r1, [r1 + r4 * 4]1591
+ lea r2, [r2 + r4 * 4]1592
+ lea r3, [r3 + r4 * 4]1593
+ PROCESS_SAD_X3_32x4_AVX5121594
+ add r0, FENC_STRIDE * 81595
+ lea r1, [r1 + r4 * 4]1596
+ lea r2, [r2 + r4 * 4]1597
+ lea r3, [r3 + r4 * 4]1598
+ PROCESS_SAD_X3_32x4_AVX5121599
+ add r0, FENC_STRIDE * 81600
+ lea r1, [r1 + r4 * 4]1601
+ lea r2, [r2 + r4 * 4]1602
+ lea r3, [r3 + r4 * 4]1603
+ PROCESS_SAD_X3_32x4_AVX5121604
+ PROCESS_SAD_X3_END_AVX5121605
+ RET1606
+1607
+;----------------------------------------------------------------------------------------------------------------------------------------1608
+; int pixel_sad_x3_48x64( const pixel* pix1, const pixel* pix2, const pixel* pix3, const pixel* pix4, intptr_t frefstride, int32_t* res )1609
+;----------------------------------------------------------------------------------------------------------------------------------------1610
+INIT_ZMM avx5121611
+cglobal pixel_sad_x3_48x64, 4, 8, 171612
+ pxor m0, m01613
+ pxor m1, m11614
+ pxor m2, m21615
+ mov r7d, 64/41616
+ vbroadcasti32x8 m16, [pw_1]1617
+1618
+ add r4d, r4d1619
+ lea r6d, [r4 * 3]1620
+.loop:1621
+ movu m4, [r0]1622
+ movu m5, [r0 + 2 * FENC_STRIDE]1623
+ movu ym6, [r0 + mmsize]1624
+ vinserti32x8 m6, [r0 + 2 * FENC_STRIDE + mmsize], 11625
+ movu m7, [r1]1626
+ movu m8, [r1 + r4]1627
+ movu ym9, [r1 + mmsize]1628
+ vinserti32x8 m9, [r1 + r4 + mmsize], 11629
+ movu m10, [r2]1630
+ movu m11, [r2 + r4]1631
+ movu ym12, [r2 + mmsize]1632
+ vinserti32x8 m12, [r2 + r4 + mmsize], 11633
+ movu m13, [r3]1634
+ movu m14, [r3 + r4]1635
+ movu ym15, [r3 + mmsize]1636
+ vinserti32x8 m15, [r3 + r4 + mmsize], 11637
+1638
+ psubw m7, m41639
+ psubw m8, m51640
+ psubw m9, m61641
+ psubw m10, m41642
+ psubw m11, m51643
+ psubw m12, m61644
+ psubw m13, m41645
+ psubw m14, m51646
+ psubw m15, m61647
+1648
+ pabsw m7, m71649
+ pabsw m8, m81650
+ pabsw m9, m91651
+ pabsw m10, m101652
+ pabsw m11, m111653
+ pabsw m12, m121654
+ pabsw m13, m131655
+ pabsw m14, m141656
+ pabsw m15, m151657
+1658
+ paddw m7, m81659
+ paddw m7, m91660
+ paddw m10, m111661
+ paddw m10, m121662
+ paddw m13, m141663
+ paddw m13, m151664
+1665
+ pmaddwd m7, m161666
+ paddd m0, m71667
+ pmaddwd m10, m161668
+ paddd m1, m101669
+ pmaddwd m13, m161670
+ paddd m2, m131671
+1672
+ movu m4, [r0 + 4 * FENC_STRIDE]1673
+ movu m5, [r0 + 6 * FENC_STRIDE]1674
+ movu ym6, [r0 + 4 * FENC_STRIDE + mmsize]1675
+ vinserti32x8 m6, [r0 + 6 * FENC_STRIDE + mmsize], 11676
+ movu m7, [r1 + 2 * r4]1677
+ movu m8, [r1 + r6]1678
+ movu ym9, [r1 + 2 * r4 + mmsize]1679
+ vinserti32x8 m9, [r1 + r6 + mmsize], 11680
+ movu m10, [r2 + 2 * r4]1681
+ movu m11, [r2 + r6]1682
+ movu ym12, [r2 + 2 * r4 + mmsize]1683
+ vinserti32x8 m12, [r2 + r6 + mmsize], 11684
+ movu m13, [r3 + 2 * r4]1685
+ movu m14, [r3 + r6]1686
+ movu ym15, [r3 + 2 * r4 + mmsize]1687
+ vinserti32x8 m15, [r3 + r6 + mmsize], 11688
+1689
+ psubw m7, m41690
+ psubw m8, m51691
+ psubw m9, m61692
+ psubw m10, m41693
+ psubw m11, m51694
+ psubw m12, m61695
+ psubw m13, m41696
+ psubw m14, m51697
+ psubw m15, m61698
+1699
+ pabsw m7, m71700
+ pabsw m8, m81701
+ pabsw m9, m91702
+ pabsw m10, m101703
+ pabsw m11, m111704
+ pabsw m12, m121705
+ pabsw m13, m131706
+ pabsw m14, m141707
+ pabsw m15, m151708
+1709
+ paddw m7, m81710
+ paddw m7, m91711
+ paddw m10, m111712
+ paddw m10, m121713
+ paddw m13, m141714
+ paddw m13, m151715
+1716
+ pmaddwd m7, m161717
+ paddd m0, m71718
+ pmaddwd m10, m161719
+ paddd m1, m101720
+ pmaddwd m13, m161721
+ paddd m2, m131722
+1723
+ add r0, FENC_STRIDE * 81724
+ lea r1, [r1 + r4 * 4]1725
+ lea r2, [r2 + r4 * 4]1726
+ lea r3, [r3 + r4 * 4]1727
+1728
+ dec r7d1729
+ jg .loop1730
+1731
+ PROCESS_SAD_X3_END_AVX5121732
+ RET1733
+%endif1734
+1735
+;------------------------------------------------------------------------------------------------------------------------------------------1736
+; void pixel_sad_x3_64x%1( const pixel* pix1, const pixel* pix2, const pixel* pix3, const pixel* pix4, intptr_t frefstride, int32_t* res )1737
+;------------------------------------------------------------------------------------------------------------------------------------------1738
+%if ARCH_X86_641739
+INIT_ZMM avx5121740
+cglobal pixel_sad_x3_64x16, 6,7,121741
+ pxor m0, m01742
+ pxor m1, m11743
+ pxor m2, m21744
+1745
+ vbroadcasti32x8 m7, [pw_1]1746
+1747
+ add r4d, r4d1748
+ lea r6d, [r4 * 3]1749
+1750
+ PROCESS_SAD_X3_64x4_AVX5121751
+ add r0, FENC_STRIDE * 81752
+ lea r1, [r1 + r4 * 4]1753
+ lea r2, [r2 + r4 * 4]1754
+ lea r3, [r3 + r4 * 4]1755
+ PROCESS_SAD_X3_64x4_AVX5121756
+ add r0, FENC_STRIDE * 81757
+ lea r1, [r1 + r4 * 4]1758
+ lea r2, [r2 + r4 * 4]1759
+ lea r3, [r3 + r4 * 4]1760
+ PROCESS_SAD_X3_64x4_AVX5121761
+ add r0, FENC_STRIDE * 81762
+ lea r1, [r1 + r4 * 4]1763
+ lea r2, [r2 + r4 * 4]1764
+ lea r3, [r3 + r4 * 4]1765
+ PROCESS_SAD_X3_64x4_AVX5121766
+ PROCESS_SAD_X3_END_AVX5121767
+ RET1768
+1769
+INIT_ZMM avx5121770
+cglobal pixel_sad_x3_64x32, 6,7,121771
+ pxor m0, m01772
+ pxor m1, m11773
+ pxor m2, m21774
+1775
+ vbroadcasti32x8 m7, [pw_1]1776
+1777
+ add r4d, r4d1778
+ lea r6d, [r4 * 3]1779
+1780
+ PROCESS_SAD_X3_64x4_AVX5121781
+ add r0, FENC_STRIDE * 81782
+ lea r1, [r1 + r4 * 4]1783
+ lea r2, [r2 + r4 * 4]1784
+ lea r3, [r3 + r4 * 4]1785
+ PROCESS_SAD_X3_64x4_AVX5121786
+ add r0, FENC_STRIDE * 81787
+ lea r1, [r1 + r4 * 4]1788
+ lea r2, [r2 + r4 * 4]1789
+ lea r3, [r3 + r4 * 4]1790
+ PROCESS_SAD_X3_64x4_AVX5121791
+ add r0, FENC_STRIDE * 81792
+ lea r1, [r1 + r4 * 4]1793
+ lea r2, [r2 + r4 * 4]1794
+ lea r3, [r3 + r4 * 4]1795
+ PROCESS_SAD_X3_64x4_AVX5121796
+ add r0, FENC_STRIDE * 81797
+ lea r1, [r1 + r4 * 4]1798
+ lea r2, [r2 + r4 * 4]1799
+ lea r3, [r3 + r4 * 4]1800
+ PROCESS_SAD_X3_64x4_AVX5121801
+ add r0, FENC_STRIDE * 81802
+ lea r1, [r1 + r4 * 4]1803
+ lea r2, [r2 + r4 * 4]1804
+ lea r3, [r3 + r4 * 4]1805
+ PROCESS_SAD_X3_64x4_AVX5121806
+ add r0, FENC_STRIDE * 81807
+ lea r1, [r1 + r4 * 4]1808
+ lea r2, [r2 + r4 * 4]1809
+ lea r3, [r3 + r4 * 4]1810
+ PROCESS_SAD_X3_64x4_AVX5121811
+ add r0, FENC_STRIDE * 81812
+ lea r1, [r1 + r4 * 4]1813
+ lea r2, [r2 + r4 * 4]1814
+ lea r3, [r3 + r4 * 4]1815
+ PROCESS_SAD_X3_64x4_AVX5121816
+ PROCESS_SAD_X3_END_AVX5121817
+ RET1818
+1819
+INIT_ZMM avx5121820
+cglobal pixel_sad_x3_64x48, 6,7,121821
+ pxor m0, m01822
+ pxor m1, m11823
+ pxor m2, m21824
+1825
+ vbroadcasti32x8 m7, [pw_1]1826
+1827
+ add r4d, r4d1828
+ lea r6d, [r4 * 3]1829
+1830
+ PROCESS_SAD_X3_64x4_AVX5121831
+ add r0, FENC_STRIDE * 81832
+ lea r1, [r1 + r4 * 4]1833
+ lea r2, [r2 + r4 * 4]1834
+ lea r3, [r3 + r4 * 4]1835
+ PROCESS_SAD_X3_64x4_AVX5121836
+ add r0, FENC_STRIDE * 81837
+ lea r1, [r1 + r4 * 4]1838
+ lea r2, [r2 + r4 * 4]1839
+ lea r3, [r3 + r4 * 4]1840
+ PROCESS_SAD_X3_64x4_AVX5121841
+ add r0, FENC_STRIDE * 81842
+ lea r1, [r1 + r4 * 4]1843
+ lea r2, [r2 + r4 * 4]1844
+ lea r3, [r3 + r4 * 4]1845
+ PROCESS_SAD_X3_64x4_AVX5121846
+ add r0, FENC_STRIDE * 81847
+ lea r1, [r1 + r4 * 4]1848
+ lea r2, [r2 + r4 * 4]1849
+ lea r3, [r3 + r4 * 4]1850
+ PROCESS_SAD_X3_64x4_AVX5121851
+ add r0, FENC_STRIDE * 81852
+ lea r1, [r1 + r4 * 4]1853
+ lea r2, [r2 + r4 * 4]1854
+ lea r3, [r3 + r4 * 4]1855
+ PROCESS_SAD_X3_64x4_AVX5121856
+ add r0, FENC_STRIDE * 81857
+ lea r1, [r1 + r4 * 4]1858
+ lea r2, [r2 + r4 * 4]1859
+ lea r3, [r3 + r4 * 4]1860
+ PROCESS_SAD_X3_64x4_AVX5121861
+ add r0, FENC_STRIDE * 81862
+ lea r1, [r1 + r4 * 4]1863
+ lea r2, [r2 + r4 * 4]1864
+ lea r3, [r3 + r4 * 4]1865
+ PROCESS_SAD_X3_64x4_AVX5121866
+ add r0, FENC_STRIDE * 81867
+ lea r1, [r1 + r4 * 4]1868
+ lea r2, [r2 + r4 * 4]1869
+ lea r3, [r3 + r4 * 4]1870
+ PROCESS_SAD_X3_64x4_AVX5121871
+ add r0, FENC_STRIDE * 81872
+ lea r1, [r1 + r4 * 4]1873
+ lea r2, [r2 + r4 * 4]1874
+ lea r3, [r3 + r4 * 4]1875
+ PROCESS_SAD_X3_64x4_AVX5121876
+ add r0, FENC_STRIDE * 81877
+ lea r1, [r1 + r4 * 4]1878
+ lea r2, [r2 + r4 * 4]1879
+ lea r3, [r3 + r4 * 4]1880
+ PROCESS_SAD_X3_64x4_AVX5121881
+ add r0, FENC_STRIDE * 81882
+ lea r1, [r1 + r4 * 4]1883
+ lea r2, [r2 + r4 * 4]1884
+ lea r3, [r3 + r4 * 4]1885
+ PROCESS_SAD_X3_64x4_AVX5121886
+ PROCESS_SAD_X3_END_AVX5121887
+ RET1888
+1889
+INIT_ZMM avx5121890
+cglobal pixel_sad_x3_64x64, 6,7,121891
+ pxor m0, m01892
+ pxor m1, m11893
+ pxor m2, m21894
+1895
+ vbroadcasti32x8 m7, [pw_1]1896
+1897
+ add r4d, r4d1898
+ lea r6d, [r4 * 3]1899
+1900
+ PROCESS_SAD_X3_64x4_AVX5121901
+ add r0, FENC_STRIDE * 81902
+ lea r1, [r1 + r4 * 4]1903
+ lea r2, [r2 + r4 * 4]1904
+ lea r3, [r3 + r4 * 4]1905
+ PROCESS_SAD_X3_64x4_AVX5121906
+ add r0, FENC_STRIDE * 81907
+ lea r1, [r1 + r4 * 4]1908
+ lea r2, [r2 + r4 * 4]1909
+ lea r3, [r3 + r4 * 4]1910
+ PROCESS_SAD_X3_64x4_AVX5121911
+ add r0, FENC_STRIDE * 81912
+ lea r1, [r1 + r4 * 4]1913
+ lea r2, [r2 + r4 * 4]1914
+ lea r3, [r3 + r4 * 4]1915
+ PROCESS_SAD_X3_64x4_AVX5121916
+ add r0, FENC_STRIDE * 81917
+ lea r1, [r1 + r4 * 4]1918
+ lea r2, [r2 + r4 * 4]1919
+ lea r3, [r3 + r4 * 4]1920
+ PROCESS_SAD_X3_64x4_AVX5121921
+ add r0, FENC_STRIDE * 81922
+ lea r1, [r1 + r4 * 4]1923
+ lea r2, [r2 + r4 * 4]1924
+ lea r3, [r3 + r4 * 4]1925
+ PROCESS_SAD_X3_64x4_AVX5121926
+ add r0, FENC_STRIDE * 81927
+ lea r1, [r1 + r4 * 4]1928
+ lea r2, [r2 + r4 * 4]1929
+ lea r3, [r3 + r4 * 4]1930
+ PROCESS_SAD_X3_64x4_AVX5121931
+ add r0, FENC_STRIDE * 81932
+ lea r1, [r1 + r4 * 4]1933
+ lea r2, [r2 + r4 * 4]1934
+ lea r3, [r3 + r4 * 4]1935
+ PROCESS_SAD_X3_64x4_AVX5121936
+ add r0, FENC_STRIDE * 81937
+ lea r1, [r1 + r4 * 4]1938
+ lea r2, [r2 + r4 * 4]1939
+ lea r3, [r3 + r4 * 4]1940
+ PROCESS_SAD_X3_64x4_AVX5121941
+ add r0, FENC_STRIDE * 81942
+ lea r1, [r1 + r4 * 4]1943
+ lea r2, [r2 + r4 * 4]1944
+ lea r3, [r3 + r4 * 4]1945
+ PROCESS_SAD_X3_64x4_AVX5121946
+ add r0, FENC_STRIDE * 81947
+ lea r1, [r1 + r4 * 4]1948
+ lea r2, [r2 + r4 * 4]1949
+ lea r3, [r3 + r4 * 4]1950
+ PROCESS_SAD_X3_64x4_AVX5121951
+ add r0, FENC_STRIDE * 81952
+ lea r1, [r1 + r4 * 4]1953
+ lea r2, [r2 + r4 * 4]1954
+ lea r3, [r3 + r4 * 4]1955
+ PROCESS_SAD_X3_64x4_AVX5121956
+ add r0, FENC_STRIDE * 81957
+ lea r1, [r1 + r4 * 4]1958
+ lea r2, [r2 + r4 * 4]1959
+ lea r3, [r3 + r4 * 4]1960
+ PROCESS_SAD_X3_64x4_AVX5121961
+ add r0, FENC_STRIDE * 81962
+ lea r1, [r1 + r4 * 4]1963
+ lea r2, [r2 + r4 * 4]1964
+ lea r3, [r3 + r4 * 4]1965
+ PROCESS_SAD_X3_64x4_AVX5121966
+ add r0, FENC_STRIDE * 81967
+ lea r1, [r1 + r4 * 4]1968
+ lea r2, [r2 + r4 * 4]1969
+ lea r3, [r3 + r4 * 4]1970
+ PROCESS_SAD_X3_64x4_AVX5121971
+ add r0, FENC_STRIDE * 81972
+ lea r1, [r1 + r4 * 4]1973
+ lea r2, [r2 + r4 * 4]1974
+ lea r3, [r3 + r4 * 4]1975
+ PROCESS_SAD_X3_64x4_AVX5121976
+ PROCESS_SAD_X3_END_AVX5121977
+ RET1978
+%endif1979
+1980
+;------------------------------------------------------------------------------------------------------------------------------------------------------------1981
+; void pixel_sad_x4_16x%1( const pixel* pix1, const pixel* pix2, const pixel* pix3, const pixel* pix4, const pixel* pix5, intptr_t frefstride, int32_t* res )1982
+;------------------------------------------------------------------------------------------------------------------------------------------------------------1983
+%if ARCH_X86_641984
+INIT_ZMM avx5121985
+cglobal pixel_sad_x4_16x8, 6,8,101986
+ pxor m0, m01987
+ pxor m1, m11988
+ pxor m2, m21989
+ pxor m3, m31990
+1991
+ vbroadcasti32x8 m9, [pw_1]1992
+1993
+ add r5d, r5d1994
+ lea r7d, [r5 * 3]1995
+1996
+ PROCESS_SAD_X4_16x4_AVX5121997
+ add r0, FENC_STRIDE * 81998
+ lea r1, [r1 + r5 * 4]1999
+ lea r2, [r2 + r5 * 4]2000
+ lea r3, [r3 + r5 * 4]2001
+ lea r4, [r4 + r5 * 4]2002
+ PROCESS_SAD_X4_16x4_AVX5122003
+ PROCESS_SAD_X4_END_AVX5122004
+ RET2005
+2006
+INIT_ZMM avx5122007
+cglobal pixel_sad_x4_16x12, 6,8,102008
+ pxor m0, m02009
+ pxor m1, m12010
+ pxor m2, m22011
+ pxor m3, m32012
+2013
+ vbroadcasti32x8 m9, [pw_1]2014
+2015
+ add r5d, r5d2016
+ lea r7d, [r5 * 3]2017
+2018
+ %rep 22019
+ PROCESS_SAD_X4_16x4_AVX5122020
+ add r0, FENC_STRIDE * 82021
+ lea r1, [r1 + r5 * 4]2022
+ lea r2, [r2 + r5 * 4]2023
+ lea r3, [r3 + r5 * 4]2024
+ lea r4, [r4 + r5 * 4]2025
+ %endrep2026
+ PROCESS_SAD_X4_16x4_AVX5122027
+ PROCESS_SAD_X4_END_AVX5122028
+ RET2029
+2030
+INIT_ZMM avx5122031
+cglobal pixel_sad_x4_16x16, 6,8,102032
+ pxor m0, m02033
+ pxor m1, m12034
+ pxor m2, m22035
+ pxor m3, m32036
+2037
+ vbroadcasti32x8 m9, [pw_1]2038
+2039
+ add r5d, r5d2040
+ lea r7d, [r5 * 3]2041
+2042
+ %rep 32043
+ PROCESS_SAD_X4_16x4_AVX5122044
+ add r0, FENC_STRIDE * 82045
+ lea r1, [r1 + r5 * 4]2046
+ lea r2, [r2 + r5 * 4]2047
+ lea r3, [r3 + r5 * 4]2048
+ lea r4, [r4 + r5 * 4]2049
+ %endrep2050
+ PROCESS_SAD_X4_16x4_AVX5122051
+ PROCESS_SAD_X4_END_AVX5122052
+ RET2053
+2054
+INIT_ZMM avx5122055
+cglobal pixel_sad_x4_16x32, 6,8,102056
+ pxor m0, m02057
+ pxor m1, m12058
+ pxor m2, m22059
+ pxor m3, m32060
+2061
+ vbroadcasti32x8 m9, [pw_1]2062
+2063
+ add r5d, r5d2064
+ lea r7d, [r5 * 3]2065
+2066
+ %rep 72067
+ PROCESS_SAD_X4_16x4_AVX5122068
+ add r0, FENC_STRIDE * 82069
+ lea r1, [r1 + r5 * 4]2070
+ lea r2, [r2 + r5 * 4]2071
+ lea r3, [r3 + r5 * 4]2072
+ lea r4, [r4 + r5 * 4]2073
+ %endrep2074
+ PROCESS_SAD_X4_16x4_AVX5122075
+ PROCESS_SAD_X4_END_AVX5122076
+ RET2077
+2078
+INIT_ZMM avx5122079
+cglobal pixel_sad_x4_16x64, 6,8,102080
+ pxor m0, m02081
+ pxor m1, m12082
+ pxor m2, m22083
+ pxor m3, m32084
+2085
+ vbroadcasti32x8 m9, [pw_1]2086
+2087
+ add r5d, r5d2088
+ lea r7d, [r5 * 3]2089
+2090
+ %rep 152091
+ PROCESS_SAD_X4_16x4_AVX5122092
+ add r0, FENC_STRIDE * 82093
+ lea r1, [r1 + r5 * 4]2094
+ lea r2, [r2 + r5 * 4]2095
+ lea r3, [r3 + r5 * 4]2096
+ lea r4, [r4 + r5 * 4]2097
+ %endrep2098
+ PROCESS_SAD_X4_16x4_AVX5122099
+ PROCESS_SAD_X4_END_AVX5122100
+ RET2101
+%endif2102
+2103
+;------------------------------------------------------------------------------------------------------------------------------------------------------------2104
+; void pixel_sad_x4_32x%1( const pixel* pix1, const pixel* pix2, const pixel* pix3, const pixel* pix4, const pixel* pix5, intptr_t frefstride, int32_t* res )2105
+;------------------------------------------------------------------------------------------------------------------------------------------------------------2106
+%if ARCH_X86_642107
+INIT_ZMM avx5122108
+cglobal pixel_sad_x4_32x8, 6,8,102109
+ pxor m0, m02110
+ pxor m1, m12111
+ pxor m2, m22112
+ pxor m3, m32113
+2114
+ vbroadcasti32x8 m9, [pw_1]2115
+2116
+ add r5d, r5d2117
+ lea r7d, [r5 * 3]2118
+2119
+ PROCESS_SAD_X4_32x4_AVX5122120
+ add r0, FENC_STRIDE * 82121
+ lea r1, [r1 + r5 * 4]2122
+ lea r2, [r2 + r5 * 4]2123
+ lea r3, [r3 + r5 * 4]2124
+ lea r4, [r4 + r5 * 4]2125
+ PROCESS_SAD_X4_32x4_AVX5122126
+ PROCESS_SAD_X4_END_AVX5122127
+ RET2128
+2129
+INIT_ZMM avx5122130
+cglobal pixel_sad_x4_32x16, 6,8,102131
+ pxor m0, m02132
+ pxor m1, m12133
+ pxor m2, m22134
+ pxor m3, m32135
+2136
+ vbroadcasti32x8 m9, [pw_1]2137
+2138
+ add r5d, r5d2139
+ lea r7d, [r5 * 3]2140
+2141
+ PROCESS_SAD_X4_32x4_AVX5122142
+ add r0, FENC_STRIDE * 82143
+ lea r1, [r1 + r5 * 4]2144
+ lea r2, [r2 + r5 * 4]2145
+ lea r3, [r3 + r5 * 4]2146
+ lea r4, [r4 + r5 * 4]2147
+ PROCESS_SAD_X4_32x4_AVX5122148
+ add r0, FENC_STRIDE * 82149
+ lea r1, [r1 + r5 * 4]2150
+ lea r2, [r2 + r5 * 4]2151
+ lea r3, [r3 + r5 * 4]2152
+ lea r4, [r4 + r5 * 4]2153
+ PROCESS_SAD_X4_32x4_AVX5122154
+ add r0, FENC_STRIDE * 82155
+ lea r1, [r1 + r5 * 4]2156
+ lea r2, [r2 + r5 * 4]2157
+ lea r3, [r3 + r5 * 4]2158
+ lea r4, [r4 + r5 * 4]2159
+ PROCESS_SAD_X4_32x4_AVX5122160
+ PROCESS_SAD_X4_END_AVX5122161
+ RET2162
+2163
+INIT_ZMM avx5122164
+cglobal pixel_sad_x4_32x24, 6,8,102165
+ pxor m0, m02166
+ pxor m1, m12167
+ pxor m2, m22168
+ pxor m3, m32169
+2170
+ vbroadcasti32x8 m9, [pw_1]2171
+2172
+ add r5d, r5d2173
+ lea r7d, [r5 * 3]2174
+2175
+ PROCESS_SAD_X4_32x4_AVX5122176
+ add r0, FENC_STRIDE * 82177
+ lea r1, [r1 + r5 * 4]2178
+ lea r2, [r2 + r5 * 4]2179
+ lea r3, [r3 + r5 * 4]2180
+ lea r4, [r4 + r5 * 4]2181
+ PROCESS_SAD_X4_32x4_AVX5122182
+ add r0, FENC_STRIDE * 82183
+ lea r1, [r1 + r5 * 4]2184
+ lea r2, [r2 + r5 * 4]2185
+ lea r3, [r3 + r5 * 4]2186
+ lea r4, [r4 + r5 * 4]2187
+ PROCESS_SAD_X4_32x4_AVX5122188
+ add r0, FENC_STRIDE * 82189
+ lea r1, [r1 + r5 * 4]2190
+ lea r2, [r2 + r5 * 4]2191
+ lea r3, [r3 + r5 * 4]2192
+ lea r4, [r4 + r5 * 4]2193
+ PROCESS_SAD_X4_32x4_AVX5122194
+ add r0, FENC_STRIDE * 82195
+ lea r1, [r1 + r5 * 4]2196
+ lea r2, [r2 + r5 * 4]2197
+ lea r3, [r3 + r5 * 4]2198
+ lea r4, [r4 + r5 * 4]2199
+ PROCESS_SAD_X4_32x4_AVX5122200
+ add r0, FENC_STRIDE * 82201
+ lea r1, [r1 + r5 * 4]2202
+ lea r2, [r2 + r5 * 4]2203
+ lea r3, [r3 + r5 * 4]2204
+ lea r4, [r4 + r5 * 4]2205
+ PROCESS_SAD_X4_32x4_AVX5122206
+ PROCESS_SAD_X4_END_AVX5122207
+ RET2208
+2209
+2210
+INIT_ZMM avx5122211
+cglobal pixel_sad_x4_32x32, 6,8,102212
+ pxor m0, m02213
+ pxor m1, m12214
+ pxor m2, m22215
+ pxor m3, m32216
+2217
+ vbroadcasti32x8 m9, [pw_1]2218
+2219
+ add r5d, r5d2220
+ lea r7d, [r5 * 3]2221
+2222
+ PROCESS_SAD_X4_32x4_AVX5122223
+ add r0, FENC_STRIDE * 82224
+ lea r1, [r1 + r5 * 4]2225
+ lea r2, [r2 + r5 * 4]2226
+ lea r3, [r3 + r5 * 4]2227
+ lea r4, [r4 + r5 * 4]2228
+ PROCESS_SAD_X4_32x4_AVX5122229
+ add r0, FENC_STRIDE * 82230
+ lea r1, [r1 + r5 * 4]2231
+ lea r2, [r2 + r5 * 4]2232
+ lea r3, [r3 + r5 * 4]2233
+ lea r4, [r4 + r5 * 4]2234
+ PROCESS_SAD_X4_32x4_AVX5122235
+ add r0, FENC_STRIDE * 82236
+ lea r1, [r1 + r5 * 4]2237
+ lea r2, [r2 + r5 * 4]2238
+ lea r3, [r3 + r5 * 4]2239
+ lea r4, [r4 + r5 * 4]2240
+ PROCESS_SAD_X4_32x4_AVX5122241
+ add r0, FENC_STRIDE * 82242
+ lea r1, [r1 + r5 * 4]2243
+ lea r2, [r2 + r5 * 4]2244
+ lea r3, [r3 + r5 * 4]2245
+ lea r4, [r4 + r5 * 4]2246
+ PROCESS_SAD_X4_32x4_AVX5122247
+ add r0, FENC_STRIDE * 82248
+ lea r1, [r1 + r5 * 4]2249
+ lea r2, [r2 + r5 * 4]2250
+ lea r3, [r3 + r5 * 4]2251
+ lea r4, [r4 + r5 * 4]2252
+ PROCESS_SAD_X4_32x4_AVX5122253
+ add r0, FENC_STRIDE * 82254
+ lea r1, [r1 + r5 * 4]2255
+ lea r2, [r2 + r5 * 4]2256
+ lea r3, [r3 + r5 * 4]2257
+ lea r4, [r4 + r5 * 4]2258
+ PROCESS_SAD_X4_32x4_AVX5122259
+ add r0, FENC_STRIDE * 82260
+ lea r1, [r1 + r5 * 4]2261
+ lea r2, [r2 + r5 * 4]2262
+ lea r3, [r3 + r5 * 4]2263
+ lea r4, [r4 + r5 * 4]2264
+ PROCESS_SAD_X4_32x4_AVX5122265
+ PROCESS_SAD_X4_END_AVX5122266
+ RET2267
+2268
+INIT_ZMM avx5122269
+cglobal pixel_sad_x4_32x64, 6,8,102270
+ pxor m0, m02271
+ pxor m1, m12272
+ pxor m2, m22273
+ pxor m3, m32274
+2275
+ vbroadcasti32x8 m9, [pw_1]2276
+2277
+ add r5d, r5d2278
+ lea r7d, [r5 * 3]2279
+2280
+ PROCESS_SAD_X4_32x4_AVX5122281
+ add r0, FENC_STRIDE * 82282
+ lea r1, [r1 + r5 * 4]2283
+ lea r2, [r2 + r5 * 4]2284
+ lea r3, [r3 + r5 * 4]2285
+ lea r4, [r4 + r5 * 4]2286
+ PROCESS_SAD_X4_32x4_AVX5122287
+ add r0, FENC_STRIDE * 82288
+ lea r1, [r1 + r5 * 4]2289
+ lea r2, [r2 + r5 * 4]2290
+ lea r3, [r3 + r5 * 4]2291
+ lea r4, [r4 + r5 * 4]2292
+ PROCESS_SAD_X4_32x4_AVX5122293
+ add r0, FENC_STRIDE * 82294
+ lea r1, [r1 + r5 * 4]2295
+ lea r2, [r2 + r5 * 4]2296
+ lea r3, [r3 + r5 * 4]2297
+ lea r4, [r4 + r5 * 4]2298
+ PROCESS_SAD_X4_32x4_AVX5122299
+ add r0, FENC_STRIDE * 82300
+ lea r1, [r1 + r5 * 4]2301
+ lea r2, [r2 + r5 * 4]2302
+ lea r3, [r3 + r5 * 4]2303
+ lea r4, [r4 + r5 * 4]2304
+ PROCESS_SAD_X4_32x4_AVX5122305
+ add r0, FENC_STRIDE * 82306
+ lea r1, [r1 + r5 * 4]2307
+ lea r2, [r2 + r5 * 4]2308
+ lea r3, [r3 + r5 * 4]2309
+ lea r4, [r4 + r5 * 4]2310
+ PROCESS_SAD_X4_32x4_AVX5122311
+ add r0, FENC_STRIDE * 82312
+ lea r1, [r1 + r5 * 4]2313
+ lea r2, [r2 + r5 * 4]2314
+ lea r3, [r3 + r5 * 4]2315
+ lea r4, [r4 + r5 * 4]2316
+ PROCESS_SAD_X4_32x4_AVX5122317
+ add r0, FENC_STRIDE * 82318
+ lea r1, [r1 + r5 * 4]2319
+ lea r2, [r2 + r5 * 4]2320
+ lea r3, [r3 + r5 * 4]2321
+ lea r4, [r4 + r5 * 4]2322
+ PROCESS_SAD_X4_32x4_AVX5122323
+ add r0, FENC_STRIDE * 82324
+ lea r1, [r1 + r5 * 4]2325
+ lea r2, [r2 + r5 * 4]2326
+ lea r3, [r3 + r5 * 4]2327
+ lea r4, [r4 + r5 * 4]2328
+ PROCESS_SAD_X4_32x4_AVX5122329
+ add r0, FENC_STRIDE * 82330
+ lea r1, [r1 + r5 * 4]2331
+ lea r2, [r2 + r5 * 4]2332
+ lea r3, [r3 + r5 * 4]2333
+ lea r4, [r4 + r5 * 4]2334
+ PROCESS_SAD_X4_32x4_AVX5122335
+ add r0, FENC_STRIDE * 82336
+ lea r1, [r1 + r5 * 4]2337
+ lea r2, [r2 + r5 * 4]2338
+ lea r3, [r3 + r5 * 4]2339
+ lea r4, [r4 + r5 * 4]2340
+ PROCESS_SAD_X4_32x4_AVX5122341
+ add r0, FENC_STRIDE * 82342
+ lea r1, [r1 + r5 * 4]2343
+ lea r2, [r2 + r5 * 4]2344
+ lea r3, [r3 + r5 * 4]2345
+ lea r4, [r4 + r5 * 4]2346
+ PROCESS_SAD_X4_32x4_AVX5122347
+ add r0, FENC_STRIDE * 82348
+ lea r1, [r1 + r5 * 4]2349
+ lea r2, [r2 + r5 * 4]2350
+ lea r3, [r3 + r5 * 4]2351
+ lea r4, [r4 + r5 * 4]2352
+ PROCESS_SAD_X4_32x4_AVX5122353
+ add r0, FENC_STRIDE * 82354
+ lea r1, [r1 + r5 * 4]2355
+ lea r2, [r2 + r5 * 4]2356
+ lea r3, [r3 + r5 * 4]2357
+ lea r4, [r4 + r5 * 4]2358
+ PROCESS_SAD_X4_32x4_AVX5122359
+ add r0, FENC_STRIDE * 82360
+ lea r1, [r1 + r5 * 4]2361
+ lea r2, [r2 + r5 * 4]2362
+ lea r3, [r3 + r5 * 4]2363
+ lea r4, [r4 + r5 * 4]2364
+ PROCESS_SAD_X4_32x4_AVX5122365
+ add r0, FENC_STRIDE * 82366
+ lea r1, [r1 + r5 * 4]2367
+ lea r2, [r2 + r5 * 4]2368
+ lea r3, [r3 + r5 * 4]2369
+ lea r4, [r4 + r5 * 4]2370
+ PROCESS_SAD_X4_32x4_AVX5122371
+ PROCESS_SAD_X4_END_AVX5122372
+ RET2373
+%endif2374
+;------------------------------------------------------------------------------------------------------------------------------------------------------------2375
+; void pixel_sad_x4_48x64( const pixel* pix1, const pixel* pix2, const pixel* pix3, const pixel* pix4, const pixel* pix5, intptr_t frefstride, int32_t* res )2376
+;------------------------------------------------------------------------------------------------------------------------------------------------------------2377
+%if ARCH_X86_642378
+INIT_ZMM avx5122379
+cglobal pixel_sad_x4_48x64, 4, 9, 202380
+ pxor m0, m02381
+ pxor m1, m12382
+ pxor m2, m22383
+ pxor m3, m32384
+ mov r8d, 64/42385
+2386
+ vbroadcasti32x8 m19, [pw_1]2387
+2388
+ add r5d, r5d2389
+ lea r7d, [r5 * 3]2390
+.loop:2391
+ movu m4, [r0]2392
+ movu m5, [r0 + 2 * FENC_STRIDE]2393
+ movu ym6, [r0 + mmsize]2394
+ vinserti32x8 m6, [r0 + 2 * FENC_STRIDE + mmsize], 12395
+ movu m7, [r1]2396
+ movu m8, [r1 + r5]2397
+ movu ym9, [r1 + mmsize]2398
+ vinserti32x8 m9, [r1 + r5 + mmsize], 12399
+ movu m10, [r2]2400
+ movu m11, [r2 + r5]2401
+ movu ym12, [r2 + mmsize]2402
+ vinserti32x8 m12, [r2 + r5 + mmsize], 12403
+ movu m13, [r3]2404
+ movu m14, [r3 + r5]2405
+ movu ym15, [r3 + mmsize]2406
+ vinserti32x8 m15, [r3 + r5 + mmsize], 12407
+ movu m16, [r4]2408
+ movu m17, [r4 + r5]2409
+ movu ym18, [r4 + mmsize]2410
+ vinserti32x8 m18, [r4 + r5 + mmsize], 12411
+2412
+ psubw m7, m42413
+ psubw m8, m52414
+ psubw m9, m62415
+ psubw m10, m42416
+ psubw m11, m52417
+ psubw m12, m62418
+ psubw m13, m42419
+ psubw m14, m52420
+ psubw m15, m62421
+ psubw m16, m42422
+ psubw m17, m52423
+ psubw m18, m62424
+2425
+ pabsw m7, m72426
+ pabsw m8, m82427
+ pabsw m9, m92428
+ pabsw m10, m102429
+ pabsw m11, m112430
+ pabsw m12, m122431
+ pabsw m13, m132432
+ pabsw m14, m142433
+ pabsw m15, m152434
+ pabsw m16, m162435
+ pabsw m17, m172436
+ pabsw m18, m182437
+2438
+ paddw m7, m82439
+ paddw m7, m92440
+ paddw m10, m112441
+ paddw m10, m122442
+ paddw m13, m142443
+ paddw m13, m152444
+ paddw m16, m172445
+ paddw m16, m182446
+2447
+ pmaddwd m7, m192448
+ paddd m0, m72449
+ pmaddwd m10, m192450
+ paddd m1, m102451
+ pmaddwd m13, m192452
+ paddd m2, m132453
+ pmaddwd m16, m192454
+ paddd m3, m162455
+2456
+ movu m4, [r0 + 4 * FENC_STRIDE]2457
+ movu m5, [r0 + 6 * FENC_STRIDE]2458
+ movu ym6, [r0 + 4 * FENC_STRIDE + mmsize]2459
+ vinserti32x8 m6, [r0 + 6 * FENC_STRIDE + mmsize], 12460
+ movu m7, [r1 + 2 * r5]2461
+ movu m8, [r1 + r7]2462
+ movu ym9, [r1 + 2 * r5 + mmsize]2463
+ vinserti32x8 m9, [r1 + r7 + mmsize], 12464
+ movu m10, [r2 + 2 * r5]2465
+ movu m11, [r2 + r7]2466
+ movu ym12, [r2 + 2 * r5 + mmsize]2467
+ vinserti32x8 m12, [r2 + r7 + mmsize], 12468
+ movu m13, [r3 + 2 * r5]2469
+ movu m14, [r3 + r7]2470
+ movu ym15, [r3 + 2 * r5 + mmsize]2471
+ vinserti32x8 m15, [r3 + r7 + mmsize], 12472
+ movu m16, [r4 + 2 * r5]2473
+ movu m17, [r4 + r7]2474
+ movu ym18, [r4 + 2 * r5 + mmsize]2475
+ vinserti32x8 m18, [r4 + r7 + mmsize], 12476
+2477
+2478
+ psubw m7, m42479
+ psubw m8, m52480
+ psubw m9, m62481
+ psubw m10, m42482
+ psubw m11, m52483
+ psubw m12, m62484
+ psubw m13, m42485
+ psubw m14, m52486
+ psubw m15, m62487
+ psubw m16, m42488
+ psubw m17, m52489
+ psubw m18, m62490
+2491
+ pabsw m7, m72492
+ pabsw m8, m82493
+ pabsw m9, m92494
+ pabsw m10, m102495
+ pabsw m11, m112496
+ pabsw m12, m122497
+ pabsw m13, m132498
+ pabsw m14, m142499
+ pabsw m15, m152500
+ pabsw m16, m162501
+ pabsw m17, m172502
+ pabsw m18, m182503
+2504
+ paddw m7, m82505
+ paddw m7, m92506
+ paddw m10, m112507
+ paddw m10, m122508
+ paddw m13, m142509
+ paddw m13, m152510
+ paddw m16, m172511
+ paddw m16, m182512
+2513
+ pmaddwd m7, m192514
+ paddd m0, m72515
+ pmaddwd m10, m192516
+ paddd m1, m102517
+ pmaddwd m13, m192518
+ paddd m2, m132519
+ pmaddwd m16, m192520
+ paddd m3, m162521
+2522
+ add r0, FENC_STRIDE * 82523
+ lea r1, [r1 + r5 * 4]2524
+ lea r2, [r2 + r5 * 4]2525
+ lea r3, [r3 + r5 * 4]2526
+ lea r4, [r4 + r5 * 4]2527
+2528
+ dec r8d2529
+ jg .loop2530
+2531
+ PROCESS_SAD_X4_END_AVX5122532
+ RET2533
+%endif2534
+2535
+;------------------------------------------------------------------------------------------------------------------------------------------------------------2536
+; void pixel_sad_x4_64x%1( const pixel* pix1, const pixel* pix2, const pixel* pix3, const pixel* pix4, const pixel* pix5, intptr_t frefstride, int32_t* res )2537
+;------------------------------------------------------------------------------------------------------------------------------------------------------------2538
+%if ARCH_X86_642539
+INIT_ZMM avx5122540
+cglobal pixel_sad_x4_64x16, 6,8,152541
+ pxor m0, m02542
+ pxor m1, m12543
+ pxor m2, m22544
+ pxor m3, m32545
+2546
+ vbroadcasti32x8 m9, [pw_1]2547
+2548
+ add r5d, r5d2549
+ lea r7d, [r5 * 3]2550
+2551
+ PROCESS_SAD_X4_64x4_AVX5122552
+ add r0, FENC_STRIDE * 82553
+ lea r1, [r1 + r5 * 4]2554
+ lea r2, [r2 + r5 * 4]2555
+ lea r3, [r3 + r5 * 4]2556
+ lea r4, [r4 + r5 * 4]2557
+ PROCESS_SAD_X4_64x4_AVX5122558
+ add r0, FENC_STRIDE * 82559
+ lea r1, [r1 + r5 * 4]2560
+ lea r2, [r2 + r5 * 4]2561
+ lea r3, [r3 + r5 * 4]2562
+ lea r4, [r4 + r5 * 4]2563
+ PROCESS_SAD_X4_64x4_AVX5122564
+ add r0, FENC_STRIDE * 82565
+ lea r1, [r1 + r5 * 4]2566
+ lea r2, [r2 + r5 * 4]2567
+ lea r3, [r3 + r5 * 4]2568
+ lea r4, [r4 + r5 * 4]2569
+ PROCESS_SAD_X4_64x4_AVX5122570
+ PROCESS_SAD_X4_END_AVX5122571
+ RET2572
+2573
+INIT_ZMM avx5122574
+cglobal pixel_sad_x4_64x32, 6,8,152575
+ pxor m0, m02576
+ pxor m1, m12577
+ pxor m2, m22578
+ pxor m3, m32579
+2580
+ vbroadcasti32x8 m9, [pw_1]2581
+2582
+ add r5d, r5d2583
+ lea r7d, [r5 * 3]2584
+2585
+ PROCESS_SAD_X4_64x4_AVX5122586
+ add r0, FENC_STRIDE * 82587
+ lea r1, [r1 + r5 * 4]2588
+ lea r2, [r2 + r5 * 4]2589
+ lea r3, [r3 + r5 * 4]2590
+ lea r4, [r4 + r5 * 4]2591
+ PROCESS_SAD_X4_64x4_AVX5122592
+ add r0, FENC_STRIDE * 82593
+ lea r1, [r1 + r5 * 4]2594
+ lea r2, [r2 + r5 * 4]2595
+ lea r3, [r3 + r5 * 4]2596
+ lea r4, [r4 + r5 * 4]2597
+ PROCESS_SAD_X4_64x4_AVX5122598
+ add r0, FENC_STRIDE * 82599
+ lea r1, [r1 + r5 * 4]2600
+ lea r2, [r2 + r5 * 4]2601
+ lea r3, [r3 + r5 * 4]2602
+ lea r4, [r4 + r5 * 4]2603
+ PROCESS_SAD_X4_64x4_AVX5122604
+ add r0, FENC_STRIDE * 82605
+ lea r1, [r1 + r5 * 4]2606
+ lea r2, [r2 + r5 * 4]2607
+ lea r3, [r3 + r5 * 4]2608
+ lea r4, [r4 + r5 * 4]2609
+ PROCESS_SAD_X4_64x4_AVX5122610
+ add r0, FENC_STRIDE * 82611
+ lea r1, [r1 + r5 * 4]2612
+ lea r2, [r2 + r5 * 4]2613
+ lea r3, [r3 + r5 * 4]2614
+ lea r4, [r4 + r5 * 4]2615
+ PROCESS_SAD_X4_64x4_AVX5122616
+ add r0, FENC_STRIDE * 82617
+ lea r1, [r1 + r5 * 4]2618
+ lea r2, [r2 + r5 * 4]2619
+ lea r3, [r3 + r5 * 4]2620
+ lea r4, [r4 + r5 * 4]2621
+ PROCESS_SAD_X4_64x4_AVX5122622
+ add r0, FENC_STRIDE * 82623
+ lea r1, [r1 + r5 * 4]2624
+ lea r2, [r2 + r5 * 4]2625
+ lea r3, [r3 + r5 * 4]2626
+ lea r4, [r4 + r5 * 4]2627
+ PROCESS_SAD_X4_64x4_AVX5122628
+ PROCESS_SAD_X4_END_AVX5122629
+ RET2630
+2631
+INIT_ZMM avx5122632
+cglobal pixel_sad_x4_64x48, 6,8,152633
+ pxor m0, m02634
+ pxor m1, m12635
+ pxor m2, m22636
+ pxor m3, m32637
+2638
+ vbroadcasti32x8 m9, [pw_1]2639
+2640
+ add r5d, r5d2641
+ lea r7d, [r5 * 3]2642
+2643
+ PROCESS_SAD_X4_64x4_AVX5122644
+ add r0, FENC_STRIDE * 82645
+ lea r1, [r1 + r5 * 4]2646
+ lea r2, [r2 + r5 * 4]2647
+ lea r3, [r3 + r5 * 4]2648
+ lea r4, [r4 + r5 * 4]2649
+ PROCESS_SAD_X4_64x4_AVX5122650
+ add r0, FENC_STRIDE * 82651
+ lea r1, [r1 + r5 * 4]2652
+ lea r2, [r2 + r5 * 4]2653
+ lea r3, [r3 + r5 * 4]2654
+ lea r4, [r4 + r5 * 4]2655
+ PROCESS_SAD_X4_64x4_AVX5122656
+ add r0, FENC_STRIDE * 82657
+ lea r1, [r1 + r5 * 4]2658
+ lea r2, [r2 + r5 * 4]2659
+ lea r3, [r3 + r5 * 4]2660
+ lea r4, [r4 + r5 * 4]2661
+ PROCESS_SAD_X4_64x4_AVX5122662
+ add r0, FENC_STRIDE * 82663
+ lea r1, [r1 + r5 * 4]2664
+ lea r2, [r2 + r5 * 4]2665
+ lea r3, [r3 + r5 * 4]2666
+ lea r4, [r4 + r5 * 4]2667
+ PROCESS_SAD_X4_64x4_AVX5122668
+ add r0, FENC_STRIDE * 82669
+ lea r1, [r1 + r5 * 4]2670
+ lea r2, [r2 + r5 * 4]2671
+ lea r3, [r3 + r5 * 4]2672
+ lea r4, [r4 + r5 * 4]2673
+ PROCESS_SAD_X4_64x4_AVX5122674
+ add r0, FENC_STRIDE * 82675
+ lea r1, [r1 + r5 * 4]2676
+ lea r2, [r2 + r5 * 4]2677
+ lea r3, [r3 + r5 * 4]2678
+ lea r4, [r4 + r5 * 4]2679
+ PROCESS_SAD_X4_64x4_AVX5122680
+ add r0, FENC_STRIDE * 82681
+ lea r1, [r1 + r5 * 4]2682
+ lea r2, [r2 + r5 * 4]2683
+ lea r3, [r3 + r5 * 4]2684
+ lea r4, [r4 + r5 * 4]2685
+ PROCESS_SAD_X4_64x4_AVX5122686
+ add r0, FENC_STRIDE * 82687
+ lea r1, [r1 + r5 * 4]2688
+ lea r2, [r2 + r5 * 4]2689
+ lea r3, [r3 + r5 * 4]2690
+ lea r4, [r4 + r5 * 4]2691
+ PROCESS_SAD_X4_64x4_AVX5122692
+ add r0, FENC_STRIDE * 82693
+ lea r1, [r1 + r5 * 4]2694
+ lea r2, [r2 + r5 * 4]2695
+ lea r3, [r3 + r5 * 4]2696
+ lea r4, [r4 + r5 * 4]2697
+ PROCESS_SAD_X4_64x4_AVX5122698
+ add r0, FENC_STRIDE * 82699
+ lea r1, [r1 + r5 * 4]2700
+ lea r2, [r2 + r5 * 4]2701
+ lea r3, [r3 + r5 * 4]2702
+ lea r4, [r4 + r5 * 4]2703
+ PROCESS_SAD_X4_64x4_AVX5122704
+ add r0, FENC_STRIDE * 82705
+ lea r1, [r1 + r5 * 4]2706
+ lea r2, [r2 + r5 * 4]2707
+ lea r3, [r3 + r5 * 4]2708
+ lea r4, [r4 + r5 * 4]2709
+ PROCESS_SAD_X4_64x4_AVX5122710
+ PROCESS_SAD_X4_END_AVX5122711
+ RET2712
+2713
+INIT_ZMM avx5122714
+cglobal pixel_sad_x4_64x64, 6,8,152715
+ pxor m0, m02716
+ pxor m1, m12717
+ pxor m2, m22718
+ pxor m3, m32719
+2720
+ vbroadcasti32x8 m9, [pw_1]2721
+2722
+ add r5d, r5d2723
+ lea r7d, [r5 * 3]2724
+2725
+ PROCESS_SAD_X4_64x4_AVX5122726
+ add r0, FENC_STRIDE * 82727
+ lea r1, [r1 + r5 * 4]2728
+ lea r2, [r2 + r5 * 4]2729
+ lea r3, [r3 + r5 * 4]2730
+ lea r4, [r4 + r5 * 4]2731
+ PROCESS_SAD_X4_64x4_AVX5122732
+ add r0, FENC_STRIDE * 82733
+ lea r1, [r1 + r5 * 4]2734
+ lea r2, [r2 + r5 * 4]2735
+ lea r3, [r3 + r5 * 4]2736
+ lea r4, [r4 + r5 * 4]2737
+ PROCESS_SAD_X4_64x4_AVX5122738
+ add r0, FENC_STRIDE * 82739
+ lea r1, [r1 + r5 * 4]2740
+ lea r2, [r2 + r5 * 4]2741
+ lea r3, [r3 + r5 * 4]2742
+ lea r4, [r4 + r5 * 4]2743
+ PROCESS_SAD_X4_64x4_AVX5122744
+ add r0, FENC_STRIDE * 82745
+ lea r1, [r1 + r5 * 4]2746
+ lea r2, [r2 + r5 * 4]2747
+ lea r3, [r3 + r5 * 4]2748
+ lea r4, [r4 + r5 * 4]2749
+ PROCESS_SAD_X4_64x4_AVX5122750
+ add r0, FENC_STRIDE * 82751
+ lea r1, [r1 + r5 * 4]2752
+ lea r2, [r2 + r5 * 4]2753
+ lea r3, [r3 + r5 * 4]2754
+ lea r4, [r4 + r5 * 4]2755
+ PROCESS_SAD_X4_64x4_AVX5122756
+ add r0, FENC_STRIDE * 82757
+ lea r1, [r1 + r5 * 4]2758
+ lea r2, [r2 + r5 * 4]2759
+ lea r3, [r3 + r5 * 4]2760
+ lea r4, [r4 + r5 * 4]2761
+ PROCESS_SAD_X4_64x4_AVX5122762
+ add r0, FENC_STRIDE * 82763
+ lea r1, [r1 + r5 * 4]2764
+ lea r2, [r2 + r5 * 4]2765
+ lea r3, [r3 + r5 * 4]2766
+ lea r4, [r4 + r5 * 4]2767
+ PROCESS_SAD_X4_64x4_AVX5122768
+ add r0, FENC_STRIDE * 82769
+ lea r1, [r1 + r5 * 4]2770
+ lea r2, [r2 + r5 * 4]2771
+ lea r3, [r3 + r5 * 4]2772
+ lea r4, [r4 + r5 * 4]2773
+ PROCESS_SAD_X4_64x4_AVX5122774
+ add r0, FENC_STRIDE * 82775
+ lea r1, [r1 + r5 * 4]2776
+ lea r2, [r2 + r5 * 4]2777
+ lea r3, [r3 + r5 * 4]2778
+ lea r4, [r4 + r5 * 4]2779
+ PROCESS_SAD_X4_64x4_AVX5122780
+ add r0, FENC_STRIDE * 82781
+ lea r1, [r1 + r5 * 4]2782
+ lea r2, [r2 + r5 * 4]2783
+ lea r3, [r3 + r5 * 4]2784
+ lea r4, [r4 + r5 * 4]2785
+ PROCESS_SAD_X4_64x4_AVX5122786
+ add r0, FENC_STRIDE * 82787
+ lea r1, [r1 + r5 * 4]2788
+ lea r2, [r2 + r5 * 4]2789
+ lea r3, [r3 + r5 * 4]2790
+ lea r4, [r4 + r5 * 4]2791
+ PROCESS_SAD_X4_64x4_AVX5122792
+ add r0, FENC_STRIDE * 82793
+ lea r1, [r1 + r5 * 4]2794
+ lea r2, [r2 + r5 * 4]2795
+ lea r3, [r3 + r5 * 4]2796
+ lea r4, [r4 + r5 * 4]2797
+ PROCESS_SAD_X4_64x4_AVX5122798
+ add r0, FENC_STRIDE * 82799
+ lea r1, [r1 + r5 * 4]2800
+ lea r2, [r2 + r5 * 4]2801
+ lea r3, [r3 + r5 * 4]2802
+ lea r4, [r4 + r5 * 4]2803
+ PROCESS_SAD_X4_64x4_AVX5122804
+ add r0, FENC_STRIDE * 82805
+ lea r1, [r1 + r5 * 4]2806
+ lea r2, [r2 + r5 * 4]2807
+ lea r3, [r3 + r5 * 4]2808
+ lea r4, [r4 + r5 * 4]2809
+ PROCESS_SAD_X4_64x4_AVX5122810
+ add r0, FENC_STRIDE * 82811
+ lea r1, [r1 + r5 * 4]2812
+ lea r2, [r2 + r5 * 4]2813
+ lea r3, [r3 + r5 * 4]2814
+ lea r4, [r4 + r5 * 4]2815
+ PROCESS_SAD_X4_64x4_AVX5122816
+ PROCESS_SAD_X4_END_AVX5122817
+ RET2818
+%endif2819
x265_2.7.tar.gz/source/common/x86/ssd-a.asm -> x265_2.9.tar.gz/source/common/x86/ssd-a.asm
Changed
590
1
2
3
; Function to find ssd for 32x16 block, sse2, 12 bit depth4
; Defined sepeartely to be called from SSD_ONE_32 macro5
+%if ARCH_X86_646
+;This code is written for 64 bit architecture7
INIT_XMM sse28
cglobal ssd_ss_32x169
pxor m8, m810
11
paddq m4, m512
paddq m9, m413
ret14
+%endif15
16
%macro SSD_ONE_32 017
+%if ARCH_X86_6418
cglobal pixel_ssd_ss_32x64, 4,7,1019
add r1d, r1d20
add r3d, r3d21
22
call ssd_ss_32x1623
movq rax, m924
RET25
+%endif26
%endmacro27
+28
%macro SSD_ONE_SS_32 029
cglobal pixel_ssd_ss_32x32, 4,5,830
add r1d, r1d31
32
RET33
%endmacro34
35
+%if ARCH_X86_6436
INIT_YMM avx237
cglobal pixel_ssd_16x16, 4,7,338
FIX_STRIDES r1, r339
40
movq rax, xm341
RET42
43
+INIT_ZMM avx51244
+cglobal pixel_ssd_32x245
+ pxor m0, m046
+ movu m1, [r0]47
+ psubw m1, [r2]48
+ pmaddwd m1, m149
+ paddd m0, m150
+ movu m1, [r0 + r1]51
+ psubw m1, [r2 + r3]52
+ pmaddwd m1, m153
+ paddd m0, m154
+ lea r0, [r0 + r1 * 2]55
+ lea r2, [r2 + r3 * 2]56
+57
+ mova m1, m058
+ pxor m2, m259
+ punpckldq m0, m260
+ punpckhdq m1, m261
+62
+ paddq m3, m063
+ paddq m3, m164
+ret65
+66
+INIT_ZMM avx51267
+cglobal pixel_ssd_32x32, 4,5,568
+ shl r1d, 169
+ shl r3d, 170
+ pxor m3, m371
+ mov r4, 1672
+.iterate:73
+ call pixel_ssd_32x274
+ dec r4d75
+ jne .iterate76
+77
+ vextracti32x8 ym4, m3, 178
+ paddq ym3, ym479
+ vextracti32x4 xm4, m3, 180
+ paddq xm3, xm481
+ movhlps xm4, xm382
+ paddq xm3, xm483
+ movq rax, xm384
+RET85
+86
+INIT_ZMM avx51287
+cglobal pixel_ssd_32x64, 4,5,588
+ shl r1d, 189
+ shl r3d, 190
+ pxor m3, m391
+ mov r4, 3292
+.iterate:93
+ call pixel_ssd_32x294
+ dec r4d95
+ jne .iterate96
+97
+ vextracti32x8 ym4, m3, 198
+ paddq ym3, ym499
+ vextracti32x4 xm4, m3, 1100
+ paddq xm3, xm4101
+ movhlps xm4, xm3102
+ paddq xm3, xm4103
+ movq rax, xm3104
+RET105
+106
+INIT_ZMM avx512107
+cglobal pixel_ssd_64x64, 4,5,5108
+ FIX_STRIDES r1, r3109
+ mov r4d, 64110
+ pxor m3, m3111
+112
+.loop:113
+ pxor m0, m0114
+ movu m1, [r0]115
+ psubw m1, [r2]116
+ pmaddwd m1, m1117
+ paddd m0, m1118
+ movu m1, [r0 + mmsize]119
+ psubw m1, [r2 + mmsize]120
+ pmaddwd m1, m1121
+ paddd m0, m1122
+123
+ lea r0, [r0 + r1]124
+ lea r2, [r2 + r3]125
+126
+ mova m1, m0127
+ pxor m2, m2128
+ punpckldq m0, m2129
+ punpckhdq m1, m2130
+ paddq m3, m0131
+ paddq m3, m1132
+133
+ dec r4d134
+ jg .loop135
+136
+ vextracti32x8 ym4, m3, 1137
+ paddq ym3, ym4138
+ vextracti32x4 xm4, m3, 1139
+ paddq xm3, xm4140
+ movhlps xm4, xm3141
+ paddq xm3, xm4142
+ movq rax, xm3143
+ RET144
+%endif145
INIT_MMX mmx2146
SSD_ONE 4, 4147
SSD_ONE 4, 8148
149
%if BIT_DEPTH <= 10150
SSD_ONE 32, 64151
SSD_ONE 32, 32152
+%if ARCH_X86_64153
SSD_TWO 64, 64154
+%endif155
%else156
SSD_ONE_32157
SSD_ONE_SS_32158
159
HADDD m2, m0160
movd eax, xm2161
RET162
+;-----------------------------------------------------------------------------163
+; ssd_ss avx512 code start164
+;-----------------------------------------------------------------------------165
+%if ARCH_X86_64166
+%macro PROCESS_SSD_SS_64x4_AVX512 0167
+ movu m0, [r0]168
+ movu m1, [r0 + mmsize]169
+ movu m2, [r0 + r1]170
+ movu m3, [r0 + r1 + mmsize]171
+ movu m4, [r2]172
+ movu m5, [r2 + mmsize]173
+ movu m6, [r2 + r3]174
+ movu m7, [r2 + r3 + mmsize]175
+176
+ psubw m0, m4177
+ psubw m1, m5178
+ psubw m2, m6179
+ psubw m3, m7180
+ pmaddwd m0, m0181
+ pmaddwd m1, m1182
+ pmaddwd m2, m2183
+ pmaddwd m3, m3184
+ paddd m8, m0185
+ paddd m8, m1186
+ paddd m8, m2187
+ paddd m8, m3188
189
+ movu m0, [r0 + 2 * r1]190
+ movu m1, [r0 + 2 * r1 + mmsize]191
+ movu m2, [r0 + r5]192
+ movu m3, [r0 + r5 + mmsize]193
+ movu m4, [r2 + 2 * r3]194
+ movu m5, [r2 + 2 * r3 + mmsize]195
+ movu m6, [r2 + r6]196
+ movu m7, [r2 + r6 + mmsize]197
+198
+ psubw m0, m4199
+ psubw m1, m5200
+ psubw m2, m6201
+ psubw m3, m7202
+ pmaddwd m0, m0203
+ pmaddwd m1, m1204
+ pmaddwd m2, m2205
+ pmaddwd m3, m3206
+ paddd m8, m0207
+ paddd m8, m1208
+ paddd m8, m2209
+ paddd m8, m3210
+%endmacro211
+212
+%macro PROCESS_SSD_SS_32x4_AVX512 0213
+ movu m0, [r0]214
+ movu m1, [r0 + r1]215
+ movu m2, [r0 + 2 * r1]216
+ movu m3, [r0 + r5]217
+ movu m4, [r2]218
+ movu m5, [r2 + r3]219
+ movu m6, [r2 + 2 * r3]220
+ movu m7, [r2 + r6]221
+222
+ psubw m0, m4223
+ psubw m1, m5224
+ psubw m2, m6225
+ psubw m3, m7226
+ pmaddwd m0, m0227
+ pmaddwd m1, m1228
+ pmaddwd m2, m2229
+ pmaddwd m3, m3230
+ paddd m8, m0231
+ paddd m8, m1232
+ paddd m8, m2233
+ paddd m8, m3234
+%endmacro235
+236
+%macro PROCESS_SSD_SS_16x4_AVX512 0237
+ movu ym0, [r0]238
+ vinserti32x8 m0, [r0 + r1], 1239
+ movu ym1, [r0 + 2 * r1]240
+ vinserti32x8 m1, [r0 + r5], 1241
+ movu ym4, [r2]242
+ vinserti32x8 m4, [r2 + r3], 1243
+ movu ym5, [r2 + 2 * r3]244
+ vinserti32x8 m5, [r2 + r6], 1245
+246
+ psubw m0, m4247
+ psubw m1, m5248
+ pmaddwd m0, m0249
+ pmaddwd m1, m1250
+ paddd m8, m0251
+ paddd m8, m1252
+%endmacro253
+254
+%macro SSD_SS_AVX512 2255
+INIT_ZMM avx512256
+cglobal pixel_ssd_ss_%1x%2, 4,7,9257
+ add r1d, r1d258
+ add r3d, r3d259
+ lea r5, [r1 * 3]260
+ lea r6, [r3 * 3]261
+ pxor m8, m8262
+263
+%rep %2/4 - 1264
+ PROCESS_SSD_SS_%1x4_AVX512265
+ lea r0, [r0 + 4 * r1]266
+ lea r2, [r2 + 4 * r3]267
+%endrep268
+ PROCESS_SSD_SS_%1x4_AVX512269
+ HADDD m8, m0270
+ movd eax, xm8271
+ RET272
+%endmacro273
+274
+275
+SSD_SS_AVX512 64, 64276
+SSD_SS_AVX512 32, 32277
+SSD_SS_AVX512 16, 16278
+%endif279
+;-----------------------------------------------------------------------------280
+; ssd_ss avx512 code end281
+;-----------------------------------------------------------------------------282
%endif ; !HIGH_BIT_DEPTH283
284
%if HIGH_BIT_DEPTH == 0285
286
movd eax, m0287
RET288
289
-290
+%if ARCH_X86_64 && BIT_DEPTH >= 10291
INIT_XMM sse2292
cglobal pixel_ssd_s_32, 2,3,5293
add r1, r1294
295
dec r2d296
jnz .loop297
298
-%if BIT_DEPTH >= 10299
movu m1, m0300
pxor m2, m2301
punpckldq m0, m2302
303
movhlps m1, m0304
paddq m0, m1305
movq rax, xm0306
-%else307
+ RET308
+%endif309
+310
+%if BIT_DEPTH == 8311
+INIT_XMM sse2312
+cglobal pixel_ssd_s_32, 2,3,5313
+ add r1, r1314
+315
+ mov r2d, 16316
+ pxor m0, m0317
+.loop:318
+ movu m1, [r0 + 0 * mmsize]319
+ movu m2, [r0 + 1 * mmsize]320
+ movu m3, [r0 + 2 * mmsize]321
+ movu m4, [r0 + 3 * mmsize]322
+ add r0, r1323
+324
+ pmaddwd m1, m1325
+ pmaddwd m2, m2326
+ pmaddwd m3, m3327
+ pmaddwd m4, m4328
+ paddd m1, m2329
+ paddd m3, m4330
+ paddd m1, m3331
+ paddd m0, m1332
+333
+ movu m1, [r0 + 0 * mmsize]334
+ movu m2, [r0 + 1 * mmsize]335
+ movu m3, [r0 + 2 * mmsize]336
+ movu m4, [r0 + 3 * mmsize]337
+ add r0, r1338
+339
+ pmaddwd m1, m1340
+ pmaddwd m2, m2341
+ pmaddwd m3, m3342
+ pmaddwd m4, m4343
+ paddd m1, m2344
+ paddd m3, m4345
+ paddd m1, m3346
+ paddd m0, m1347
+348
+ dec r2d349
+ jnz .loop350
; calculate sum and return351
HADDD m0, m1352
movd eax, m0353
-%endif354
RET355
+%endif356
357
+%if ARCH_X86_64358
INIT_YMM avx2359
cglobal pixel_ssd_s_16, 2,4,5360
add r1, r1361
362
movd eax, xm0363
%endif364
RET365
+%endif366
+;-----------------------------------------------------------------------------367
+; ssd_s avx512 code start368
+;-----------------------------------------------------------------------------369
+%macro PROCESS_SSD_S_32x8_AVX512 0370
+ movu m1, [r0]371
+ movu m2, [r0 + r1]372
+ movu m3, [r0 + 2 * r1]373
+ movu m4, [r0 + r3]374
+375
+ pmaddwd m1, m1376
+ pmaddwd m2, m2377
+ pmaddwd m3, m3378
+ pmaddwd m4, m4379
+ paddd m1, m2380
+ paddd m3, m4381
+ paddd m1, m3382
+ paddd m0, m1383
+384
+ lea r0, [r0 + 4 * r1]385
+386
+ movu m1, [r0]387
+ movu m2, [r0 + r1]388
+ movu m3, [r0 + 2 * r1]389
+ movu m4, [r0 + r3]390
+391
+ pmaddwd m1, m1392
+ pmaddwd m2, m2393
+ pmaddwd m3, m3394
+ pmaddwd m4, m4395
+ paddd m1, m2396
+ paddd m3, m4397
+ paddd m1, m3398
+ paddd m0, m1399
+%endmacro400
+401
+%macro PROCESS_SSD_S_16x8_AVX512 0402
+ movu ym1, [r0]403
+ vinserti32x8 m1, [r0 + r1], 1404
+ movu ym2, [r0 + 2 * r1]405
+ vinserti32x8 m2, [r0 + r3], 1406
+ lea r0, [r0 + 4 * r1]407
+ movu ym3, [r0]408
+ vinserti32x8 m3, [r0 + r1], 1409
+ movu ym4, [r0 + 2 * r1]410
+ vinserti32x8 m4, [r0 + r3], 1411
+ pmaddwd m1, m1412
+ pmaddwd m2, m2413
+ pmaddwd m3, m3414
+ pmaddwd m4, m4415
+ paddd m1, m2416
+ paddd m3, m4417
+ paddd m1, m3418
+ paddd m0, m1419
+%endmacro420
+;-----------------------------------------------------------------------------421
+; int pixel_ssd_s( int16_t *ref, intptr_t i_stride )422
+;-----------------------------------------------------------------------------423
+%if ARCH_X86_64424
+INIT_ZMM avx512425
+cglobal pixel_ssd_s_32, 2,4,5426
+ add r1, r1427
+ lea r3, [r1 * 3]428
+ pxor m0, m0429
+430
+ PROCESS_SSD_S_32x8_AVX512431
+ lea r0, [r0 + 4 * r1]432
+ PROCESS_SSD_S_32x8_AVX512433
+ lea r0, [r0 + 4 * r1]434
+ PROCESS_SSD_S_32x8_AVX512435
+ lea r0, [r0 + 4 * r1]436
+ PROCESS_SSD_S_32x8_AVX512437
+438
+ ; calculate sum and return439
+%if BIT_DEPTH >= 10440
+ movu m1, m0441
+ pxor m2, m2442
+ punpckldq m0, m2443
+ punpckhdq m1, m2444
+ paddq m0, m1445
+ vextracti32x8 ym2, m0, 1446
+ paddq ym0, ym2447
+ vextracti32x4 xm2, m0, 1448
+ paddq xm2, xm0449
+ movhlps xm1, xm2450
+ paddq xm2, xm1451
+ movq rax, xm2452
+%else453
+ HADDD m0, m1454
+ movd eax, xm0455
+%endif456
+ RET457
+458
+INIT_ZMM avx512459
+cglobal pixel_ssd_s_16, 2,4,5460
+ add r1, r1461
+ lea r3, [r1 * 3]462
+ pxor m0, m0463
+464
+ PROCESS_SSD_S_16x8_AVX512465
+ lea r0, [r0 + 4 * r1]466
+ PROCESS_SSD_S_16x8_AVX512467
+468
+ ; calculate sum and return469
+ HADDD m0, m1470
+ movd eax, xm0471
+ RET472
+%endif473
+;-----------------------------------------------------------------------------474
+; ssd_s avx512 code end475
+;-----------------------------------------------------------------------------476
+;-----------------------------------------------------------------------------477
+;ALigned version of macro478
+;-----------------------------------------------------------------------------479
+%macro PROCESS_SSD_S_16x8_ALIGNED_AVX512 0480
+ mova ym1, [r0]481
+ vinserti32x8 m1, [r0 + r1], 1482
+ mova ym2, [r0 + 2 * r1]483
+ vinserti32x8 m2, [r0 + r3], 1484
+ lea r0, [r0 + 4 * r1]485
+ mova ym3, [r0]486
+ vinserti32x8 m3, [r0 + r1], 1487
+ mova ym4, [r0 + 2 * r1]488
+ vinserti32x8 m4, [r0 + r3], 1489
+ pmaddwd m1, m1490
+ pmaddwd m2, m2491
+ pmaddwd m3, m3492
+ pmaddwd m4, m4493
+ paddd m1, m2494
+ paddd m3, m4495
+ paddd m1, m3496
+ paddd m0, m1497
+%endmacro498
+;---------------------------------------------------------------------------------499
+;int pixel_ssd_s_aligned( int16_t *ref, intptr_t i_stride )500
+;-----------------------------------------------------------------------------------501
+%if ARCH_X86_64502
+INIT_ZMM avx512503
+504
+INIT_ZMM avx512505
+cglobal pixel_ssd_s_aligned_16, 2,4,5506
+ add r1, r1507
+ lea r3, [r1 * 3]508
+ pxor m0, m0509
+510
+ PROCESS_SSD_S_16x8_ALIGNED_AVX512511
+ lea r0, [r0 + 4 * r1]512
+ PROCESS_SSD_S_16x8_ALIGNED_AVX512513
+514
+ ; calculate sum and return515
+ HADDD m0, m1516
+ movd eax, xm0517
+ RET518
+%endif519
+;---------------------------------------------------------------------------------------------520
+; aligned implementation for 32521
+;---------------------------------------------------------------------------------------------522
+%macro PROCESS_SSD_S_32x8_ALIGNED_AVX512 0523
+ mova m1, [r0]524
+ mova m2, [r0 + r1]525
+ mova m3, [r0 + 2 * r1]526
+ mova m4, [r0 + r3]527
+528
+ pmaddwd m1, m1529
+ pmaddwd m2, m2530
+ pmaddwd m3, m3531
+ pmaddwd m4, m4532
+ paddd m1, m2533
+ paddd m3, m4534
+ paddd m1, m3535
+ paddd m0, m1536
+537
+ lea r0, [r0 + 4 * r1]538
+539
+ mova m1, [r0]540
+ mova m2, [r0 + r1]541
+ mova m3, [r0 + 2 * r1]542
+ mova m4, [r0 + r3]543
+544
+ pmaddwd m1, m1545
+ pmaddwd m2, m2546
+ pmaddwd m3, m3547
+ pmaddwd m4, m4548
+ paddd m1, m2549
+ paddd m3, m4550
+ paddd m1, m3551
+ paddd m0, m1552
+%endmacro553
+554
+%if ARCH_X86_64555
+INIT_ZMM avx512556
+cglobal pixel_ssd_s_aligned_32, 2,4,5557
+ add r1, r1558
+ lea r3, [r1 * 3]559
+ pxor m0, m0560
+561
+ PROCESS_SSD_S_32x8_AVX512562
+ lea r0, [r0 + 4 * r1]563
+ PROCESS_SSD_S_32x8_ALIGNED_AVX512564
+ lea r0, [r0 + 4 * r1]565
+ PROCESS_SSD_S_32x8_ALIGNED_AVX512566
+ lea r0, [r0 + 4 * r1]567
+ PROCESS_SSD_S_32x8_ALIGNED_AVX512568
+569
+ ; calculate sum and return570
+%if BIT_DEPTH >= 10571
+ mova m1, m0572
+ pxor m2, m2573
+ punpckldq m0, m2574
+ punpckhdq m1, m2575
+ paddq m0, m1576
+ vextracti32x8 ym2, m0, 1577
+ paddq ym0, ym2578
+ vextracti32x4 xm2, m0, 1579
+ paddq xm2, xm0580
+ movhlps xm1, xm2581
+ paddq xm2, xm1582
+ movq rax, xm2583
+%else584
+ HADDD m0, m1585
+ movd eax, xm0586
+%endif587
+ RET588
+%endif589
\ No newline at end of file590
x265_2.7.tar.gz/source/common/x86/v4-ipfilter16.asm -> x265_2.9.tar.gz/source/common/x86/v4-ipfilter16.asm
Changed
17
1
2
RET3
%endmacro4
5
+%if ARCH_X86_646
FILTER_VER_CHROMA_AVX2_4xN pp, 16, 1, 67
FILTER_VER_CHROMA_AVX2_4xN ps, 16, 0, INTERP_SHIFT_PS8
FILTER_VER_CHROMA_AVX2_4xN sp, 16, 1, INTERP_SHIFT_SP9
10
FILTER_VER_CHROMA_AVX2_4xN ps, 32, 0, INTERP_SHIFT_PS11
FILTER_VER_CHROMA_AVX2_4xN sp, 32, 1, INTERP_SHIFT_SP12
FILTER_VER_CHROMA_AVX2_4xN ss, 32, 0, 613
+%endif14
15
%macro FILTER_VER_CHROMA_AVX2_8x8 316
INIT_YMM avx217
x265_2.7.tar.gz/source/common/x86/v4-ipfilter8.asm -> x265_2.9.tar.gz/source/common/x86/v4-ipfilter8.asm
Changed
359
1
2
const v4_interp4_vpp_shuf1, dd 0, 1, 1, 2, 2, 3, 3, 43
dd 2, 3, 3, 4, 4, 5, 5, 64
5
-const tab_ChromaCoeff, db 0, 64, 0, 06
+const v4_tab_ChromaCoeff, db 0, 64, 0, 07
db -2, 58, 10, -28
db -4, 54, 16, -29
db -6, 46, 28, -410
11
mova m6, [r5 + r4]12
mova m5, [r5 + r4 + 16]13
%else14
- mova m6, [tab_ChromaCoeff + r4]15
- mova m5, [tab_ChromaCoeff + r4 + 16]16
+ mova m6, [v4_tab_ChromaCoeff + r4]17
+ mova m5, [v4_tab_ChromaCoeff + r4 + 16]18
%endif19
20
%ifidn %1,pp21
22
sub r0, r123
24
%ifdef PIC25
- lea r5, [tab_ChromaCoeff]26
+ lea r5, [v4_tab_ChromaCoeff]27
movd m0, [r5 + r4 * 4]28
%else29
- movd m0, [tab_ChromaCoeff + r4 * 4]30
+ movd m0, [v4_tab_ChromaCoeff + r4 * 4]31
%endif32
lea r4, [r1 * 3]33
lea r5, [r0 + 4 * r1]34
35
sub r0, r136
37
%ifdef PIC38
- lea r5, [tab_ChromaCoeff]39
+ lea r5, [v4_tab_ChromaCoeff]40
movd m0, [r5 + r4 * 4]41
%else42
- movd m0, [tab_ChromaCoeff + r4 * 4]43
+ movd m0, [v4_tab_ChromaCoeff + r4 * 4]44
%endif45
46
pshufb m0, [tab_Cm]47
48
sub r0, r149
50
%ifdef PIC51
- lea r5, [tab_ChromaCoeff]52
+ lea r5, [v4_tab_ChromaCoeff]53
movd m0, [r5 + r4 * 4]54
%else55
- movd m0, [tab_ChromaCoeff + r4 * 4]56
+ movd m0, [v4_tab_ChromaCoeff + r4 * 4]57
%endif58
59
pshufb m0, [tab_Cm]60
61
sub r0, r162
63
%ifdef PIC64
- lea r5, [tab_ChromaCoeff]65
+ lea r5, [v4_tab_ChromaCoeff]66
movd m0, [r5 + r4 * 4]67
%else68
- movd m0, [tab_ChromaCoeff + r4 * 4]69
+ movd m0, [v4_tab_ChromaCoeff + r4 * 4]70
%endif71
72
pshufb m0, [tab_Cm]73
74
sub r0, r175
76
%ifdef PIC77
- lea r5, [tab_ChromaCoeff]78
+ lea r5, [v4_tab_ChromaCoeff]79
movd m0, [r5 + r4 * 4]80
%else81
- movd m0, [tab_ChromaCoeff + r4 * 4]82
+ movd m0, [v4_tab_ChromaCoeff + r4 * 4]83
%endif84
85
pshufb m0, [tab_Cm]86
87
punpcklbw m4, m2, m388
89
%ifdef PIC90
- lea r6, [tab_ChromaCoeff]91
+ lea r6, [v4_tab_ChromaCoeff]92
movd m5, [r6 + r4 * 4]93
%else94
- movd m5, [tab_ChromaCoeff + r4 * 4]95
+ movd m5, [v4_tab_ChromaCoeff + r4 * 4]96
%endif97
98
pshufb m6, m5, [tab_Vm]99
100
add r3d, r3d101
102
%ifdef PIC103
- lea r5, [tab_ChromaCoeff]104
+ lea r5, [v4_tab_ChromaCoeff]105
movd m0, [r5 + r4 * 4]106
%else107
- movd m0, [tab_ChromaCoeff + r4 * 4]108
+ movd m0, [v4_tab_ChromaCoeff + r4 * 4]109
%endif110
111
pshufb m0, [tab_Cm]112
113
add r3d, r3d114
115
%ifdef PIC116
- lea r5, [tab_ChromaCoeff]117
+ lea r5, [v4_tab_ChromaCoeff]118
movd m0, [r5 + r4 * 4]119
%else120
- movd m0, [tab_ChromaCoeff + r4 * 4]121
+ movd m0, [v4_tab_ChromaCoeff + r4 * 4]122
%endif123
124
pshufb m0, [tab_Cm]125
126
add r3d, r3d127
128
%ifdef PIC129
- lea r5, [tab_ChromaCoeff]130
+ lea r5, [v4_tab_ChromaCoeff]131
movd m0, [r5 + r4 * 4]132
%else133
- movd m0, [tab_ChromaCoeff + r4 * 4]134
+ movd m0, [v4_tab_ChromaCoeff + r4 * 4]135
%endif136
137
pshufb m0, [tab_Cm]138
139
add r3d, r3d140
141
%ifdef PIC142
- lea r5, [tab_ChromaCoeff]143
+ lea r5, [v4_tab_ChromaCoeff]144
movd m5, [r5 + r4 * 4]145
%else146
- movd m5, [tab_ChromaCoeff + r4 * 4]147
+ movd m5, [v4_tab_ChromaCoeff + r4 * 4]148
%endif149
150
pshufb m6, m5, [tab_Vm]151
152
add r3d, r3d153
154
%ifdef PIC155
- lea r5, [tab_ChromaCoeff]156
+ lea r5, [v4_tab_ChromaCoeff]157
movd m5, [r5 + r4 * 4]158
%else159
- movd m5, [tab_ChromaCoeff + r4 * 4]160
+ movd m5, [v4_tab_ChromaCoeff + r4 * 4]161
%endif162
163
pshufb m6, m5, [tab_Vm]164
165
add r3d, r3d166
167
%ifdef PIC168
- lea r5, [tab_ChromaCoeff]169
+ lea r5, [v4_tab_ChromaCoeff]170
movd m5, [r5 + r4 * 4]171
%else172
- movd m5, [tab_ChromaCoeff + r4 * 4]173
+ movd m5, [v4_tab_ChromaCoeff + r4 * 4]174
%endif175
176
pshufb m6, m5, [tab_Vm]177
178
add r3d, r3d179
180
%ifdef PIC181
- lea r5, [tab_ChromaCoeff]182
+ lea r5, [v4_tab_ChromaCoeff]183
movd m0, [r5 + r4 * 4]184
%else185
- movd m0, [tab_ChromaCoeff + r4 * 4]186
+ movd m0, [v4_tab_ChromaCoeff + r4 * 4]187
%endif188
189
pshufb m1, m0, [tab_Vm]190
191
add r3d, r3d192
193
%ifdef PIC194
- lea r5, [tab_ChromaCoeff]195
+ lea r5, [v4_tab_ChromaCoeff]196
movd m0, [r5 + r4 * 4]197
%else198
- movd m0, [tab_ChromaCoeff + r4 * 4]199
+ movd m0, [v4_tab_ChromaCoeff + r4 * 4]200
%endif201
202
pshufb m1, m0, [tab_Vm]203
204
add r3d, r3d205
206
%ifdef PIC207
- lea r5, [tab_ChromaCoeff]208
+ lea r5, [v4_tab_ChromaCoeff]209
movd m0, [r5 + r4 * 4]210
%else211
- movd m0, [tab_ChromaCoeff + r4 * 4]212
+ movd m0, [v4_tab_ChromaCoeff + r4 * 4]213
%endif214
215
pshufb m1, m0, [tab_Vm]216
217
add r3d, r3d218
219
%ifdef PIC220
- lea r5, [tab_ChromaCoeff]221
+ lea r5, [v4_tab_ChromaCoeff]222
movd m0, [r5 + r4 * 4]223
%else224
- movd m0, [tab_ChromaCoeff + r4 * 4]225
+ movd m0, [v4_tab_ChromaCoeff + r4 * 4]226
%endif227
228
pshufb m1, m0, [tab_Vm]229
230
sub r0, r1231
232
%ifdef PIC233
- lea r5, [tab_ChromaCoeff]234
+ lea r5, [v4_tab_ChromaCoeff]235
movd m5, [r5 + r4 * 4]236
%else237
- movd m5, [tab_ChromaCoeff + r4 * 4]238
+ movd m5, [v4_tab_ChromaCoeff + r4 * 4]239
%endif240
241
pshufb m6, m5, [tab_Vm]242
243
sub r0, r1244
245
%ifdef PIC246
- lea r5, [tab_ChromaCoeff]247
+ lea r5, [v4_tab_ChromaCoeff]248
movd m5, [r5 + r4 * 4]249
%else250
- movd m5, [tab_ChromaCoeff + r4 * 4]251
+ movd m5, [v4_tab_ChromaCoeff + r4 * 4]252
%endif253
254
pshufb m6, m5, [tab_Vm]255
256
sub r0, r1257
258
%ifdef PIC259
- lea r5, [tab_ChromaCoeff]260
+ lea r5, [v4_tab_ChromaCoeff]261
movd m0, [r5 + r4 * 4]262
%else263
- movd m0, [tab_ChromaCoeff + r4 * 4]264
+ movd m0, [v4_tab_ChromaCoeff + r4 * 4]265
%endif266
267
pshufb m1, m0, [tab_Vm]268
269
sub r0, r1270
271
%ifdef PIC272
- lea r5, [tab_ChromaCoeff]273
+ lea r5, [v4_tab_ChromaCoeff]274
movd m0, [r5 + r4 * 4]275
%else276
- movd m0, [tab_ChromaCoeff + r4 * 4]277
+ movd m0, [v4_tab_ChromaCoeff + r4 * 4]278
%endif279
280
pshufb m1, m0, [tab_Vm]281
282
sub r0, r1283
284
%ifdef PIC285
- lea r5, [tab_ChromaCoeff]286
+ lea r5, [v4_tab_ChromaCoeff]287
movd m0, [r5 + r4 * 4]288
%else289
- movd m0, [tab_ChromaCoeff + r4 * 4]290
+ movd m0, [v4_tab_ChromaCoeff + r4 * 4]291
%endif292
293
pshufb m1, m0, [tab_Vm]294
295
sub r0, r1296
297
%ifdef PIC298
- lea r5, [tab_ChromaCoeff]299
+ lea r5, [v4_tab_ChromaCoeff]300
movd m0, [r5 + r4 * 4]301
%else302
- movd m0, [tab_ChromaCoeff + r4 * 4]303
+ movd m0, [v4_tab_ChromaCoeff + r4 * 4]304
%endif305
306
pshufb m1, m0, [tab_Vm]307
308
sub r0, r1309
310
%ifdef PIC311
- lea r5, [tab_ChromaCoeff]312
+ lea r5, [v4_tab_ChromaCoeff]313
movd m0, [r5 + r4 * 4]314
%else315
- movd m0, [tab_ChromaCoeff + r4 * 4]316
+ movd m0, [v4_tab_ChromaCoeff + r4 * 4]317
%endif318
319
pshufb m1, m0, [tab_Vm]320
321
add r3d, r3d322
323
%ifdef PIC324
- lea r5, [tab_ChromaCoeff]325
+ lea r5, [v4_tab_ChromaCoeff]326
movd m0, [r5 + r4 * 4]327
%else328
- movd m0, [tab_ChromaCoeff + r4 * 4]329
+ movd m0, [v4_tab_ChromaCoeff + r4 * 4]330
%endif331
332
pshufb m1, m0, [tab_Vm]333
334
add r3d, r3d335
336
%ifdef PIC337
- lea r5, [tab_ChromaCoeff]338
+ lea r5, [v4_tab_ChromaCoeff]339
movd m0, [r5 + r4 * 4]340
%else341
- movd m0, [tab_ChromaCoeff + r4 * 4]342
+ movd m0, [v4_tab_ChromaCoeff + r4 * 4]343
%endif344
345
pshufb m0, [tab_Cm]346
347
add r3d, r3d348
349
%ifdef PIC350
- lea r5, [tab_ChromaCoeff]351
+ lea r5, [v4_tab_ChromaCoeff]352
movd m0, [r5 + r4 * 4]353
%else354
- movd m0, [tab_ChromaCoeff + r4 * 4]355
+ movd m0, [v4_tab_ChromaCoeff + r4 * 4]356
%endif357
358
pshufb m0, [tab_Cm]359
x265_2.7.tar.gz/source/common/x86/x86inc.asm -> x265_2.9.tar.gz/source/common/x86/x86inc.asm
Changed
502
1
2
%endif3
4
%macro SECTION_RODATA 0-1 325
- SECTION .rodata align=%16
+ %ifidn __OUTPUT_FORMAT__,win327
+ SECTION .rdata align=%18
+ %elif WIN649
+ SECTION .rdata align=%110
+ %else11
+ SECTION .rodata align=%112
+ %endif13
%endmacro14
15
%if WIN6416
17
%endmacro18
19
%define required_stack_alignment ((mmsize + 15) & ~15)20
+%define vzeroupper_required (mmsize > 16 && (ARCH_X86_64 == 0 || xmm_regs_used > 16 || notcpuflag(avx512)))21
+%define high_mm_regs (16*cpuflag(avx512))22
23
%macro ALLOC_STACK 1-2 0 ; stack_size, n_xmm_regs (for win64 only)24
%ifnum %125
26
27
%macro WIN64_PUSH_XMM 028
; Use the shadow space to store XMM6 and XMM7, the rest needs stack space allocated.29
- %if xmm_regs_used > 630
+ %if xmm_regs_used > 6 + high_mm_regs31
movaps [rstk + stack_offset + 8], xmm632
%endif33
- %if xmm_regs_used > 734
+ %if xmm_regs_used > 7 + high_mm_regs35
movaps [rstk + stack_offset + 24], xmm736
%endif37
- %if xmm_regs_used > 838
+ %assign %%xmm_regs_on_stack xmm_regs_used - high_mm_regs - 839
+ %if %%xmm_regs_on_stack > 040
%assign %%i 841
- %rep xmm_regs_used-842
+ %rep %%xmm_regs_on_stack43
movaps [rsp + (%%i-8)*16 + stack_size + 32], xmm %+ %%i44
%assign %%i %%i+145
%endrep46
47
48
%macro WIN64_SPILL_XMM 149
%assign xmm_regs_used %150
- ASSERT xmm_regs_used <= 1651
- %if xmm_regs_used > 852
+ ASSERT xmm_regs_used <= 16 + high_mm_regs53
+ %assign %%xmm_regs_on_stack xmm_regs_used - high_mm_regs - 854
+ %if %%xmm_regs_on_stack > 055
; Allocate stack space for callee-saved xmm registers plus shadow space and align the stack.56
%assign %%pad (xmm_regs_used-8)*16 + 3257
%assign stack_size_padded %%pad + ((-%%pad-stack_offset-gprsize) & (STACK_ALIGNMENT-1))58
59
60
%macro WIN64_RESTORE_XMM_INTERNAL 061
%assign %%pad_size 062
- %if xmm_regs_used > 863
- %assign %%i xmm_regs_used64
- %rep xmm_regs_used-865
+ %assign %%xmm_regs_on_stack xmm_regs_used - high_mm_regs - 866
+ %if %%xmm_regs_on_stack > 067
+ %assign %%i xmm_regs_used - high_mm_regs68
+ %rep %%xmm_regs_on_stack69
%assign %%i %%i-170
movaps xmm %+ %%i, [rsp + (%%i-8)*16 + stack_size + 32]71
%endrep72
73
%assign %%pad_size stack_size_padded74
%endif75
%endif76
- %if xmm_regs_used > 777
+ %if xmm_regs_used > 7 + high_mm_regs78
movaps xmm7, [rsp + stack_offset - %%pad_size + 24]79
%endif80
- %if xmm_regs_used > 681
+ %if xmm_regs_used > 6 + high_mm_regs82
movaps xmm6, [rsp + stack_offset - %%pad_size + 8]83
%endif84
%endmacro85
86
%assign xmm_regs_used 087
%endmacro88
89
-%define has_epilogue regs_used > 7 || xmm_regs_used > 6 || mmsize == 32 || stack_size > 090
+%define has_epilogue regs_used > 7 || stack_size > 0 || vzeroupper_required || xmm_regs_used > 6 + high_mm_regs91
92
%macro RET 093
WIN64_RESTORE_XMM_INTERNAL94
POP_IF_USED 14, 13, 12, 11, 10, 9, 8, 795
- %if mmsize == 3296
+ %if vzeroupper_required97
vzeroupper98
%endif99
AUTO_REP_RET100
101
DECLARE_REG 13, R12, 64102
DECLARE_REG 14, R13, 72103
104
-%macro PROLOGUE 2-5+ ; #args, #regs, #xmm_regs, [stack_size,] arg_names...105
+%macro PROLOGUE 2-5+ 0; #args, #regs, #xmm_regs, [stack_size,] arg_names...106
%assign num_args %1107
%assign regs_used %2108
+ %assign xmm_regs_used %3109
ASSERT regs_used >= num_args110
SETUP_STACK_POINTER %4111
ASSERT regs_used <= 15112
113
DEFINE_ARGS_INTERNAL %0, %4, %5114
%endmacro115
116
-%define has_epilogue regs_used > 9 || mmsize == 32 || stack_size > 0117
+%define has_epilogue regs_used > 9 || stack_size > 0 || vzeroupper_required118
119
%macro RET 0120
%if stack_size_padded > 0121
122
%endif123
%endif124
POP_IF_USED 14, 13, 12, 11, 10, 9125
- %if mmsize == 32126
+ %if vzeroupper_required127
vzeroupper128
%endif129
AUTO_REP_RET130
131
DEFINE_ARGS_INTERNAL %0, %4, %5132
%endmacro133
134
-%define has_epilogue regs_used > 3 || mmsize == 32 || stack_size > 0135
+%define has_epilogue regs_used > 3 || stack_size > 0 || vzeroupper_required136
137
%macro RET 0138
%if stack_size_padded > 0139
140
%endif141
%endif142
POP_IF_USED 6, 5, 4, 3143
- %if mmsize == 32144
+ %if vzeroupper_required145
vzeroupper146
%endif147
AUTO_REP_RET148
149
%assign stack_offset 0 ; stack pointer offset relative to the return address150
%assign stack_size 0 ; amount of stack space that can be freely used inside a function151
%assign stack_size_padded 0 ; total amount of allocated stack space, including space for callee-saved xmm registers on WIN64 and alignment padding152
- %assign xmm_regs_used 0 ; number of XMM registers requested, used for dealing with callee-saved registers on WIN64153
+ %assign xmm_regs_used 0 ; number of XMM registers requested, used for dealing with callee-saved registers on WIN64 and vzeroupper154
%ifnidn %3, ""155
PROLOGUE %3156
%endif157
%endmacro158
159
+; Create a global symbol from a local label with the correct name mangling and type160
+%macro cglobal_label 1161
+ %if FORMAT_ELF162
+ global current_function %+ %1:function hidden163
+ %else164
+ global current_function %+ %1165
+ %endif166
+ %1:167
+%endmacro168
+169
%macro cextern 1170
%xdefine %1 mangle(private_prefix %+ _ %+ %1)171
CAT_XDEFINE cglobaled_, %1, 1172
173
%assign cpuflags_bmi1 (1<<16)| cpuflags_avx | cpuflags_lzcnt174
%assign cpuflags_bmi2 (1<<17)| cpuflags_bmi1175
%assign cpuflags_avx2 (1<<18)| cpuflags_fma3 | cpuflags_bmi2176
+%assign cpuflags_avx512 (1<<19)| cpuflags_avx2 ; F, CD, BW, DQ, VL177
178
-%assign cpuflags_cache32 (1<<19)179
-%assign cpuflags_cache64 (1<<20)180
-%assign cpuflags_slowctz (1<<21)181
+%assign cpuflags_cache32 (1<<20)182
+%assign cpuflags_cache64 (1<<21)183
%assign cpuflags_aligned (1<<22) ; not a cpu feature, but a function variant184
%assign cpuflags_atom (1<<23)185
186
187
%endif188
%endmacro189
190
-; Merge mmx and sse*191
+; Merge mmx and sse*, and avx*192
; m# is a simd register of the currently selected size193
; xm# is the corresponding xmm register if mmsize >= 16, otherwise the same as m#194
; ym# is the corresponding ymm register if mmsize >= 32, otherwise the same as m#195
-; (All 3 remain in sync through SWAP.)196
+; zm# is the corresponding zmm register if mmsize >= 64, otherwise the same as m#197
+; (All 4 remain in sync through SWAP.)198
199
%macro CAT_XDEFINE 3200
%xdefine %1%2 %3201
202
%undef %1%2203
%endmacro204
205
+%macro DEFINE_MMREGS 1 ; mmtype206
+ %assign %%prev_mmregs 0207
+ %ifdef num_mmregs208
+ %assign %%prev_mmregs num_mmregs209
+ %endif210
+211
+ %assign num_mmregs 8212
+ %if ARCH_X86_64 && mmsize >= 16213
+ %assign num_mmregs 16214
+ %if cpuflag(avx512) || mmsize == 64215
+ %assign num_mmregs 32216
+ %endif217
+ %endif218
+219
+ %assign %%i 0220
+ %rep num_mmregs221
+ CAT_XDEFINE m, %%i, %1 %+ %%i222
+ CAT_XDEFINE nn%1, %%i, %%i223
+ %assign %%i %%i+1224
+ %endrep225
+ %if %%prev_mmregs > num_mmregs226
+ %rep %%prev_mmregs - num_mmregs227
+ CAT_UNDEF m, %%i228
+ CAT_UNDEF nn %+ mmtype, %%i229
+ %assign %%i %%i+1230
+ %endrep231
+ %endif232
+ %xdefine mmtype %1233
+%endmacro234
+235
+; Prefer registers 16-31 over 0-15 to avoid having to use vzeroupper236
+%macro AVX512_MM_PERMUTATION 0-1 0 ; start_reg237
+ %if ARCH_X86_64 && cpuflag(avx512)238
+ %assign %%i %1239
+ %rep 16-%1240
+ %assign %%i_high %%i+16241
+ SWAP %%i, %%i_high242
+ %assign %%i %%i+1243
+ %endrep244
+ %endif245
+%endmacro246
+247
%macro INIT_MMX 0-1+248
%assign avx_enabled 0249
%define RESET_MM_PERMUTATION INIT_MMX %1250
%define mmsize 8251
- %define num_mmregs 8252
%define mova movq253
%define movu movq254
%define movh movd255
%define movnta movntq256
- %assign %%i 0257
- %rep 8258
- CAT_XDEFINE m, %%i, mm %+ %%i259
- CAT_XDEFINE nnmm, %%i, %%i260
- %assign %%i %%i+1261
- %endrep262
- %rep 8263
- CAT_UNDEF m, %%i264
- CAT_UNDEF nnmm, %%i265
- %assign %%i %%i+1266
- %endrep267
INIT_CPUFLAGS %1268
+ DEFINE_MMREGS mm269
%endmacro270
271
%macro INIT_XMM 0-1+272
%assign avx_enabled 0273
%define RESET_MM_PERMUTATION INIT_XMM %1274
%define mmsize 16275
- %define num_mmregs 8276
- %if ARCH_X86_64277
- %define num_mmregs 16278
- %endif279
%define mova movdqa280
%define movu movdqu281
%define movh movq282
%define movnta movntdq283
- %assign %%i 0284
- %rep num_mmregs285
- CAT_XDEFINE m, %%i, xmm %+ %%i286
- CAT_XDEFINE nnxmm, %%i, %%i287
- %assign %%i %%i+1288
- %endrep289
INIT_CPUFLAGS %1290
+ DEFINE_MMREGS xmm291
+ %if WIN64292
+ ; Swap callee-saved registers with volatile registers293
+ AVX512_MM_PERMUTATION 6294
+ %endif295
%endmacro296
297
%macro INIT_YMM 0-1+298
%assign avx_enabled 1299
%define RESET_MM_PERMUTATION INIT_YMM %1300
%define mmsize 32301
- %define num_mmregs 8302
- %if ARCH_X86_64303
- %define num_mmregs 16304
- %endif305
%define mova movdqa306
%define movu movdqu307
%undef movh308
%define movnta movntdq309
- %assign %%i 0310
- %rep num_mmregs311
- CAT_XDEFINE m, %%i, ymm %+ %%i312
- CAT_XDEFINE nnymm, %%i, %%i313
- %assign %%i %%i+1314
- %endrep315
INIT_CPUFLAGS %1316
+ DEFINE_MMREGS ymm317
+ AVX512_MM_PERMUTATION318
+%endmacro319
+320
+%macro INIT_ZMM 0-1+321
+ %assign avx_enabled 1322
+ %define RESET_MM_PERMUTATION INIT_ZMM %1323
+ %define mmsize 64324
+ %define mova movdqa325
+ %define movu movdqu326
+ %undef movh327
+ %define movnta movntdq328
+ INIT_CPUFLAGS %1329
+ DEFINE_MMREGS zmm330
+ AVX512_MM_PERMUTATION331
%endmacro332
333
INIT_XMM334
335
%define mmmm%1 mm%1336
%define mmxmm%1 mm%1337
%define mmymm%1 mm%1338
+ %define mmzmm%1 mm%1339
%define xmmmm%1 mm%1340
%define xmmxmm%1 xmm%1341
%define xmmymm%1 xmm%1342
+ %define xmmzmm%1 xmm%1343
%define ymmmm%1 mm%1344
%define ymmxmm%1 xmm%1345
%define ymmymm%1 ymm%1346
+ %define ymmzmm%1 ymm%1347
+ %define zmmmm%1 mm%1348
+ %define zmmxmm%1 xmm%1349
+ %define zmmymm%1 ymm%1350
+ %define zmmzmm%1 zmm%1351
%define xm%1 xmm %+ m%1352
%define ym%1 ymm %+ m%1353
+ %define zm%1 zmm %+ m%1354
%endmacro355
356
%assign i 0357
-%rep 16358
+%rep 32359
DECLARE_MMCAST i360
%assign i i+1361
%endrep362
363
;=============================================================================364
365
%assign i 0366
-%rep 16367
+%rep 32368
%if i < 8369
CAT_XDEFINE sizeofmm, i, 8370
+ CAT_XDEFINE regnumofmm, i, i371
%endif372
CAT_XDEFINE sizeofxmm, i, 16373
CAT_XDEFINE sizeofymm, i, 32374
+ CAT_XDEFINE sizeofzmm, i, 64375
+ CAT_XDEFINE regnumofxmm, i, i376
+ CAT_XDEFINE regnumofymm, i, i377
+ CAT_XDEFINE regnumofzmm, i, i378
%assign i i+1379
%endrep380
%undef i381
382
%endmacro383
%endmacro384
385
-; Instructions with both VEX and non-VEX encodings386
+; Instructions with both VEX/EVEX and legacy encodings387
; Non-destructive instructions are written without parameters388
AVX_INSTR addpd, sse2, 1, 0, 1389
AVX_INSTR addps, sse, 1, 0, 1390
391
AVX_INSTR addss, sse, 1, 0, 0392
AVX_INSTR addsubpd, sse3, 1, 0, 0393
AVX_INSTR addsubps, sse3, 1, 0, 0394
-AVX_INSTR aesdec, fnord, 0, 0, 0395
-AVX_INSTR aesdeclast, fnord, 0, 0, 0396
-AVX_INSTR aesenc, fnord, 0, 0, 0397
-AVX_INSTR aesenclast, fnord, 0, 0, 0398
-AVX_INSTR aesimc399
-AVX_INSTR aeskeygenassist400
+AVX_INSTR aesdec, aesni, 0, 0, 0401
+AVX_INSTR aesdeclast, aesni, 0, 0, 0402
+AVX_INSTR aesenc, aesni, 0, 0, 0403
+AVX_INSTR aesenclast, aesni, 0, 0, 0404
+AVX_INSTR aesimc, aesni405
+AVX_INSTR aeskeygenassist, aesni406
AVX_INSTR andnpd, sse2, 1, 0, 0407
AVX_INSTR andnps, sse, 1, 0, 0408
AVX_INSTR andpd, sse2, 1, 0, 1409
410
AVX_INSTR blendps, sse4, 1, 1, 0411
AVX_INSTR blendvpd, sse4 ; can't be emulated412
AVX_INSTR blendvps, sse4 ; can't be emulated413
+AVX_INSTR cmpeqpd, sse2, 1, 0, 1414
+AVX_INSTR cmpeqps, sse, 1, 0, 1415
+AVX_INSTR cmpeqsd, sse2, 1, 0, 0416
+AVX_INSTR cmpeqss, sse, 1, 0, 0417
+AVX_INSTR cmplepd, sse2, 1, 0, 0418
+AVX_INSTR cmpleps, sse, 1, 0, 0419
+AVX_INSTR cmplesd, sse2, 1, 0, 0420
+AVX_INSTR cmpless, sse, 1, 0, 0421
+AVX_INSTR cmpltpd, sse2, 1, 0, 0422
+AVX_INSTR cmpltps, sse, 1, 0, 0423
+AVX_INSTR cmpltsd, sse2, 1, 0, 0424
+AVX_INSTR cmpltss, sse, 1, 0, 0425
+AVX_INSTR cmpneqpd, sse2, 1, 0, 1426
+AVX_INSTR cmpneqps, sse, 1, 0, 1427
+AVX_INSTR cmpneqsd, sse2, 1, 0, 0428
+AVX_INSTR cmpneqss, sse, 1, 0, 0429
+AVX_INSTR cmpnlepd, sse2, 1, 0, 0430
+AVX_INSTR cmpnleps, sse, 1, 0, 0431
+AVX_INSTR cmpnlesd, sse2, 1, 0, 0432
+AVX_INSTR cmpnless, sse, 1, 0, 0433
+AVX_INSTR cmpnltpd, sse2, 1, 0, 0434
+AVX_INSTR cmpnltps, sse, 1, 0, 0435
+AVX_INSTR cmpnltsd, sse2, 1, 0, 0436
+AVX_INSTR cmpnltss, sse, 1, 0, 0437
+AVX_INSTR cmpordpd, sse2 1, 0, 1438
+AVX_INSTR cmpordps, sse 1, 0, 1439
+AVX_INSTR cmpordsd, sse2 1, 0, 0440
+AVX_INSTR cmpordss, sse 1, 0, 0441
AVX_INSTR cmppd, sse2, 1, 1, 0442
AVX_INSTR cmpps, sse, 1, 1, 0443
AVX_INSTR cmpsd, sse2, 1, 1, 0444
AVX_INSTR cmpss, sse, 1, 1, 0445
+AVX_INSTR cmpunordpd, sse2, 1, 0, 1446
+AVX_INSTR cmpunordps, sse, 1, 0, 1447
+AVX_INSTR cmpunordsd, sse2, 1, 0, 0448
+AVX_INSTR cmpunordss, sse, 1, 0, 0449
AVX_INSTR comisd, sse2450
AVX_INSTR comiss, sse451
AVX_INSTR cvtdq2pd, sse2452
453
FMA4_INSTR fmsubadd, pd, ps454
FMA4_INSTR fnmadd, pd, ps, sd, ss455
FMA4_INSTR fnmsub, pd, ps, sd, ss456
+457
+; Macros for converting VEX instructions to equivalent EVEX ones.458
+%macro EVEX_INSTR 2-3 0 ; vex, evex, prefer_evex459
+ %macro %1 2-7 fnord, fnord, %1, %2, %3460
+ %ifidn %3, fnord461
+ %define %%args %1, %2462
+ %elifidn %4, fnord463
+ %define %%args %1, %2, %3464
+ %else465
+ %define %%args %1, %2, %3, %4466
+ %endif467
+ %assign %%evex_required cpuflag(avx512) & %7468
+ %ifnum regnumof%1469
+ %if regnumof%1 >= 16 || sizeof%1 > 32470
+ %assign %%evex_required 1471
+ %endif472
+ %endif473
+ %ifnum regnumof%2474
+ %if regnumof%2 >= 16 || sizeof%2 > 32475
+ %assign %%evex_required 1476
+ %endif477
+ %endif478
+ %if %%evex_required479
+ %6 %%args480
+ %else481
+ %5 %%args ; Prefer VEX over EVEX due to shorter instruction length482
+ %endif483
+ %endmacro484
+%endmacro485
+486
+EVEX_INSTR vbroadcastf128, vbroadcastf32x4487
+EVEX_INSTR vbroadcasti128, vbroadcasti32x4488
+EVEX_INSTR vextractf128, vextractf32x4489
+EVEX_INSTR vextracti128, vextracti32x4490
+EVEX_INSTR vinsertf128, vinsertf32x4491
+EVEX_INSTR vinserti128, vinserti32x4492
+EVEX_INSTR vmovdqa, vmovdqa32493
+EVEX_INSTR vmovdqu, vmovdqu32494
+EVEX_INSTR vpand, vpandd495
+EVEX_INSTR vpandn, vpandnd496
+EVEX_INSTR vpor, vpord497
+EVEX_INSTR vpxor, vpxord498
+EVEX_INSTR vrcpps, vrcp14ps, 1 ; EVEX versions have higher precision499
+EVEX_INSTR vrcpss, vrcp14ss, 1500
+EVEX_INSTR vrsqrtps, vrsqrt14ps, 1501
+EVEX_INSTR vrsqrtss, vrsqrt14ss, 1502
x265_2.7.tar.gz/source/common/x86/x86util.asm -> x265_2.9.tar.gz/source/common/x86/x86util.asm
Changed
101
1
2
pminsw %2, %43
%endmacro4
5
+%macro MOVHL 2 ; dst, src6
+%ifidn %1, %27
+ punpckhqdq %1, %28
+%elif cpuflag(avx)9
+ punpckhqdq %1, %2, %210
+%elif cpuflag(sse4)11
+ pshufd %1, %2, q3232 ; pshufd is slow on some older CPUs, so only use it on more modern ones12
+%else13
+ movhlps %1, %2 ; may cause an int/float domain transition and has a dependency on dst14
+%endif15
+%endmacro16
+17
%macro HADDD 2 ; sum junk18
-%if sizeof%1 == 3219
-%define %2 xmm%220
- vextracti128 %2, %1, 121
-%define %1 xmm%122
- paddd %1, %223
+%if sizeof%1 >= 6424
+ vextracti32x8 ymm%2, zmm%1, 125
+ paddd ymm%1, ymm%226
%endif27
-%if mmsize >= 1628
-%if cpuflag(xop) && sizeof%1 == 1629
- vphadddq %1, %130
+%if sizeof%1 >= 3231
+ vextracti128 xmm%2, ymm%1, 132
+ paddd xmm%1, xmm%233
+%endif34
+%if sizeof%1 >= 1635
+ MOVHL xmm%2, xmm%136
+ paddd xmm%1, xmm%237
%endif38
- movhlps %2, %139
- paddd %1, %240
+%if cpuflag(xop) && sizeof%1 == 1641
+ vphadddq xmm%1, xmm%142
%endif43
%if notcpuflag(xop)44
- PSHUFLW %2, %1, q003245
- paddd %1, %246
+ PSHUFLW xmm%2, xmm%1, q103247
+ paddd xmm%1, xmm%248
%endif49
-%undef %150
-%undef %251
%endmacro52
53
%macro HADDW 2 ; reg, tmp54
%if cpuflag(xop) && sizeof%1 == 1655
vphaddwq %1, %156
- movhlps %2, %157
+ MOVHL %2, %158
paddd %1, %259
%else60
pmaddwd %1, [pw_1]61
62
%macro HADDUW 263
%if cpuflag(xop) && sizeof%1 == 1664
vphadduwq %1, %165
- movhlps %2, %166
+ MOVHL %2, %167
paddd %1, %268
%else69
HADDUWD %1, %270
71
%if %6 ; %5 aligned?72
mova %1, %473
psubw %1, %574
+%elif cpuflag(avx)75
+ movu %1, %476
+ psubw %1, %577
%else78
movu %1, %479
movu %2, %580
psubw %1, %281
%endif82
%else ; !HIGH_BIT_DEPTH83
-%ifidn %3, none84
movh %1, %485
movh %2, %586
+%ifidn %3, none87
punpcklbw %1, %288
punpcklbw %2, %289
- psubw %1, %290
%else91
- movh %1, %492
punpcklbw %1, %393
- movh %2, %594
punpcklbw %2, %395
- psubw %1, %296
%endif97
+ psubw %1, %298
%endif ; HIGH_BIT_DEPTH99
%endmacro100
101
x265_2.7.tar.gz/source/common/yuv.cpp -> x265_2.9.tar.gz/source/common/yuv.cpp
Changed
39
1
2
3
void Yuv::addClip(const Yuv& srcYuv0, const ShortYuv& srcYuv1, uint32_t log2SizeL, int picCsp)4
{5
- primitives.cu[log2SizeL - 2].add_ps(m_buf[0], m_size, srcYuv0.m_buf[0], srcYuv1.m_buf[0], srcYuv0.m_size, srcYuv1.m_size);6
+ primitives.cu[log2SizeL - 2].add_ps[(m_size % 64 == 0) && (srcYuv0.m_size % 64 == 0) && (srcYuv1.m_size % 64 == 0)](m_buf[0],7
+ m_size, srcYuv0.m_buf[0], srcYuv1.m_buf[0], srcYuv0.m_size, srcYuv1.m_size);8
if (m_csp != X265_CSP_I400 && picCsp != X265_CSP_I400)9
{10
- primitives.chroma[m_csp].cu[log2SizeL - 2].add_ps(m_buf[1], m_csize, srcYuv0.m_buf[1], srcYuv1.m_buf[1], srcYuv0.m_csize, srcYuv1.m_csize);11
- primitives.chroma[m_csp].cu[log2SizeL - 2].add_ps(m_buf[2], m_csize, srcYuv0.m_buf[2], srcYuv1.m_buf[2], srcYuv0.m_csize, srcYuv1.m_csize);12
+ primitives.chroma[m_csp].cu[log2SizeL - 2].add_ps[(m_csize % 64 == 0) && (srcYuv0.m_csize % 64 ==0) && (srcYuv1.m_csize % 64 == 0)](m_buf[1],13
+ m_csize, srcYuv0.m_buf[1], srcYuv1.m_buf[1], srcYuv0.m_csize, srcYuv1.m_csize);14
+ primitives.chroma[m_csp].cu[log2SizeL - 2].add_ps[(m_csize % 64 == 0) && (srcYuv0.m_csize % 64 == 0) && (srcYuv1.m_csize % 64 == 0)](m_buf[2],15
+ m_csize, srcYuv0.m_buf[2], srcYuv1.m_buf[2], srcYuv0.m_csize, srcYuv1.m_csize);16
}17
if (picCsp == X265_CSP_I400 && m_csp != X265_CSP_I400)18
{19
20
const int16_t* srcY0 = srcYuv0.getLumaAddr(absPartIdx);21
const int16_t* srcY1 = srcYuv1.getLumaAddr(absPartIdx);22
pixel* dstY = getLumaAddr(absPartIdx);23
- primitives.pu[part].addAvg(srcY0, srcY1, dstY, srcYuv0.m_size, srcYuv1.m_size, m_size);24
+ primitives.pu[part].addAvg[(srcYuv0.m_size % 64 == 0) && (srcYuv1.m_size % 64 == 0) && (m_size % 64 == 0)](srcY0, srcY1, dstY, srcYuv0.m_size, srcYuv1.m_size, m_size);25
}26
if (bChroma)27
{28
29
const int16_t* srcV1 = srcYuv1.getCrAddr(absPartIdx);30
pixel* dstU = getCbAddr(absPartIdx);31
pixel* dstV = getCrAddr(absPartIdx);32
- primitives.chroma[m_csp].pu[part].addAvg(srcU0, srcU1, dstU, srcYuv0.m_csize, srcYuv1.m_csize, m_csize);33
- primitives.chroma[m_csp].pu[part].addAvg(srcV0, srcV1, dstV, srcYuv0.m_csize, srcYuv1.m_csize, m_csize);34
+ primitives.chroma[m_csp].pu[part].addAvg[(srcYuv0.m_csize % 64 == 0) && (srcYuv1.m_csize % 64 == 0) && (m_csize % 64 == 0)](srcU0, srcU1, dstU, srcYuv0.m_csize, srcYuv1.m_csize, m_csize);35
+ primitives.chroma[m_csp].pu[part].addAvg[(srcYuv0.m_csize % 64 == 0) && (srcYuv1.m_csize % 64 == 0) && (m_csize % 64 == 0)](srcV0, srcV1, dstV, srcYuv0.m_csize, srcYuv1.m_csize, m_csize);36
}37
}38
39
x265_2.7.tar.gz/source/common/yuv.h -> x265_2.9.tar.gz/source/common/yuv.h
Changed
9
1
2
class Yuv3
{4
public:5
-6
pixel* m_buf[3];7
8
uint32_t m_size;9
x265_2.7.tar.gz/source/dynamicHDR10/SeiMetadataDictionary.cpp -> x265_2.9.tar.gz/source/dynamicHDR10/SeiMetadataDictionary.cpp
Changed
28
1
2
const std::string BezierCurveNames::NumberOfAnchors = std::string("NumberOfAnchors");3
const std::string BezierCurveNames::KneePointX = std::string("KneePointX");4
const std::string BezierCurveNames::KneePointY = std::string("KneePointY");5
+const std::string BezierCurveNames::AnchorsTag = std::string("Anchors");6
const std::string BezierCurveNames::Anchors[] = {std::string("Anchor0"),7
std::string("Anchor1"),8
std::string("Anchor2"),9
10
11
const std::string PercentileNames::TagName = std::string("PercentileLuminance");12
const std::string PercentileNames::NumberOfPercentiles = std::string("NumberOfPercentiles");13
+const std::string PercentileNames::DistributionIndex = std::string("DistributionIndex");14
+const std::string PercentileNames::DistributionValues = std::string("DistributionValues");15
const std::string PercentileNames::PercentilePercentageValue[] = {std::string("PercentilePercentage0"),16
std::string("PercentilePercentage1"),17
std::string("PercentilePercentage2"),18
19
20
21
const std::string LuminanceNames::TagName = std::string("LuminanceParameters");22
+const std::string LuminanceNames::LlcTagName = std::string("LuminanceDistributions");23
const std::string LuminanceNames::AverageRGB = std::string("AverageRGB");24
+const std::string LuminanceNames::MaxSCL = std::string("MaxScl");25
const std::string LuminanceNames::MaxSCL0 = std::string("MaxScl0");26
const std::string LuminanceNames::MaxSCL1 = std::string("MaxScl1");27
const std::string LuminanceNames::MaxSCL2 = std::string("MaxScl2");28
x265_2.7.tar.gz/source/dynamicHDR10/SeiMetadataDictionary.h -> x265_2.9.tar.gz/source/dynamicHDR10/SeiMetadataDictionary.h
Changed
28
1
2
static const std::string NumberOfAnchors;3
static const std::string KneePointX;4
static const std::string KneePointY;5
+ static const std::string AnchorsTag;6
static const std::string Anchors[14];7
};8
//Ellipse Selection Data9
10
public:11
static const std::string TagName;12
static const std::string NumberOfPercentiles;13
+ static const std::string DistributionIndex;14
+ static const std::string DistributionValues;15
static const std::string PercentilePercentageValue[15];16
static const std::string PercentileLuminanceValue[15];17
};18
19
{20
public:21
static const std::string TagName;22
+ static const std::string LlcTagName;23
static const std::string AverageRGB;24
+ static const std::string MaxSCL;25
static const std::string MaxSCL0;26
static const std::string MaxSCL1;27
static const std::string MaxSCL2;28
x265_2.7.tar.gz/source/dynamicHDR10/metadataFromJson.cpp -> x265_2.9.tar.gz/source/dynamicHDR10/metadataFromJson.cpp
Changed
534
1
2
int mCurrentStreamBit;3
int mCurrentStreamByte;4
5
- bool luminanceParamFromJson(const Json &data, LuminanceParameters &obj)6
+ bool luminanceParamFromJson(const Json &data, LuminanceParameters &obj, const JsonType jsonType)7
{8
JsonObject lumJsonData = data.object_items();9
if(!lumJsonData.empty())10
{11
- JsonObject percentileData = lumJsonData[PercentileNames::TagName].object_items();12
- obj.order = percentileData[PercentileNames::NumberOfPercentiles].int_value();13
-14
- obj.averageLuminance = static_cast<float>(lumJsonData[LuminanceNames::AverageRGB].number_value());15
- obj.maxRLuminance = static_cast<float>(lumJsonData[LuminanceNames::MaxSCL0].number_value());16
- obj.maxGLuminance = static_cast<float>(lumJsonData[LuminanceNames::MaxSCL1].number_value());17
- obj.maxBLuminance = static_cast<float>(lumJsonData[LuminanceNames::MaxSCL2].number_value());18
-19
- if(!percentileData.empty())20
- {21
- obj.percentiles.resize(obj.order);22
- for(int i = 0; i < obj.order; ++i)23
- {24
- std::string percentileTag = PercentileNames::TagName;25
- percentileTag += std::to_string(i);26
- obj.percentiles[i] = static_cast<unsigned int>(percentileData[percentileTag].int_value());27
- }28
- }29
-30
- return true;31
- }32
- return false;33
- }34
-35
- bool percentagesFromJson(const Json &data, std::vector<unsigned int> &percentages)36
- {37
- JsonObject jsonData = data.object_items();38
- if(!jsonData.empty())39
- {40
- JsonObject percentileData = jsonData[PercentileNames::TagName].object_items();41
- int order = percentileData[PercentileNames::NumberOfPercentiles].int_value();42
-43
- percentages.resize(order);44
- for(int i = 0; i < order; ++i)45
- {46
- std::string percentileTag = PercentileNames::PercentilePercentageValue[i];47
- percentages[i] = static_cast<unsigned int>(percentileData[percentileTag].int_value());48
- }49
-50
- return true;51
- }52
+ switch(jsonType)53
+ {54
+ case LEGACY:55
+ {56
+ obj.averageLuminance = static_cast<float>(lumJsonData[LuminanceNames::AverageRGB].number_value());57
+ obj.maxRLuminance = static_cast<float>(lumJsonData[LuminanceNames::MaxSCL0].number_value());58
+ obj.maxGLuminance = static_cast<float>(lumJsonData[LuminanceNames::MaxSCL1].number_value());59
+ obj.maxBLuminance = static_cast<float>(lumJsonData[LuminanceNames::MaxSCL2].number_value());60
+61
+ JsonObject percentileData = lumJsonData[PercentileNames::TagName].object_items();62
+ obj.order = percentileData[PercentileNames::NumberOfPercentiles].int_value();63
+ if(!percentileData.empty())64
+ {65
+ obj.percentiles.resize(obj.order);66
+ for(int i = 0; i < obj.order; ++i)67
+ {68
+ std::string percentileTag = PercentileNames::TagName;69
+ percentileTag += std::to_string(i);70
+ obj.percentiles[i] = static_cast<unsigned int>(percentileData[percentileTag].int_value());71
+ }72
+ }73
+ return true;74
+ } break;75
+ case LLC:76
+ {77
+ obj.averageLuminance = static_cast<float>(lumJsonData[LuminanceNames::AverageRGB].number_value());78
+ JsonArray maxScl = lumJsonData[LuminanceNames::MaxSCL].array_items();79
+ obj.maxRLuminance = static_cast<float>(maxScl[0].number_value());80
+ obj.maxGLuminance = static_cast<float>(maxScl[1].number_value());81
+ obj.maxBLuminance = static_cast<float>(maxScl[2].number_value());82
+83
+ JsonObject percentileData = lumJsonData[LuminanceNames::LlcTagName].object_items();84
+ if(!percentileData.empty())85
+ {86
+ JsonArray distributionValues = percentileData[PercentileNames::DistributionValues].array_items();87
+ obj.order = static_cast<int>(distributionValues.size());88
+ obj.percentiles.resize(obj.order);89
+ for(int i = 0; i < obj.order; ++i)90
+ {91
+ obj.percentiles[i] = static_cast<unsigned int>(distributionValues[i].int_value());92
+ }93
+ }94
+ return true;95
+ } break;96
+ }97
+ }98
return false;99
}100
101
- bool percentagesFromJson(const Json &data, unsigned int *percentages)102
+ bool percentagesFromJson(const Json &data, std::vector<unsigned int> &percentages, const JsonType jsonType)103
{104
JsonObject jsonData = data.object_items();105
if(!jsonData.empty())106
{107
- JsonObject percentileData = jsonData[PercentileNames::TagName].object_items();108
- int order = percentileData[PercentileNames::NumberOfPercentiles].int_value();109
-110
- for(int i = 0; i < order; ++i)111
- {112
- std::string percentileTag = PercentileNames::PercentilePercentageValue[i];113
- percentages[i] = static_cast<unsigned int>(percentileData[percentileTag].int_value());114
- }115
+ switch(jsonType)116
+ {117
+ case LEGACY:118
+ {119
+ JsonObject percentileData = jsonData[PercentileNames::TagName].object_items();120
+ int order = percentileData[PercentileNames::NumberOfPercentiles].int_value();121
+ percentages.resize(order);122
+ for(int i = 0; i < order; ++i)123
+ {124
+ std::string percentileTag = PercentileNames::PercentilePercentageValue[i];125
+ percentages[i] = static_cast<unsigned int>(percentileData[percentileTag].int_value());126
+ }127
+ return true;128
+ } break;129
+ case LLC:130
+ {131
+ JsonObject percentileData = jsonData[LuminanceNames::LlcTagName].object_items();132
+ if(!percentileData.empty())133
+ {134
+ JsonArray percentageValues = percentileData[PercentileNames::DistributionIndex].array_items();135
+ int order = static_cast<int>(percentageValues.size());136
+ percentages.resize(order);137
+ for(int i = 0; i < order; ++i)138
+ {139
+ percentages[i] = static_cast<unsigned int>(percentageValues[i].int_value());140
+ }141
+ } 142
+ return true;143
+ } break;144
+ }145
146
- return true;147
}148
return false;149
}150
151
- bool bezierCurveFromJson(const Json &data, BezierCurveData &obj)152
+ bool bezierCurveFromJson(const Json &data, BezierCurveData &obj, const JsonType jsonType)153
{154
JsonObject jsonData = data.object_items();155
if(!jsonData.empty())156
{157
- obj.order = jsonData[BezierCurveNames::NumberOfAnchors].int_value();158
- obj.coeff.resize(obj.order);159
- obj.sPx = jsonData[BezierCurveNames::KneePointX].int_value();160
- obj.sPy = jsonData[BezierCurveNames::KneePointY].int_value();161
- for(int i = 0; i < obj.order; ++i)162
- {163
- obj.coeff[i] = jsonData[BezierCurveNames::Anchors[i]].int_value();164
- }165
-166
- return true;167
+ switch(jsonType)168
+ {169
+ case LEGACY:170
+ {171
+ obj.sPx = jsonData[BezierCurveNames::KneePointX].int_value();172
+ obj.sPy = jsonData[BezierCurveNames::KneePointY].int_value();173
+ obj.order = jsonData[BezierCurveNames::NumberOfAnchors].int_value();174
+ obj.coeff.resize(obj.order);175
+ for(int i = 0; i < obj.order; ++i)176
+ {177
+ obj.coeff[i] = jsonData[BezierCurveNames::Anchors[i]].int_value();178
+ }179
+ return true; 180
+ } break;181
+ case LLC:182
+ {183
+ obj.sPx = jsonData[BezierCurveNames::KneePointX].int_value();184
+ obj.sPy = jsonData[BezierCurveNames::KneePointY].int_value();185
+ JsonArray anchorValues = data[BezierCurveNames::AnchorsTag].array_items();186
+ obj.order = static_cast<int>(anchorValues.size());187
+ obj.coeff.resize(obj.order);188
+ for(int i = 0; i < obj.order; ++i)189
+ {190
+ obj.coeff[i] = anchorValues[i].int_value();191
+ }192
+ return true;193
+ } break;194
+ }195
}196
return false;197
}198
199
void setPayloadSize(uint8_t *dataStream, int positionOnStream, int payload)200
{201
int payloadBytes = 1;202
-203
for(;payload >= 0xFF; payload -= 0xFF, ++payloadBytes);204
-205
if(payloadBytes > 1)206
{207
shiftData(dataStream, payloadBytes-1, mCurrentStreamByte, positionOnStream);208
209
}210
}211
212
-// const std::string LocalParameters = std::string("LocalParameters");213
-// const std::string TargetDisplayLuminance = std::string("TargetedSystemDisplayMaximumLuminance");214
};215
216
metadataFromJson::metadataFromJson() :217
218
delete mPimpl;219
}220
221
-222
bool metadataFromJson::frameMetadataFromJson(const char* filePath,223
int frame,224
uint8_t *&metadata)225
{226
std::string path(filePath);227
JsonArray fileData = JsonHelper::readJsonArray(path);228
-229
+ JsonType jsonType = LEGACY;230
if(fileData.empty())231
{232
- return false;233
+ jsonType = LLC;234
+ fileData = JsonHelper::readJson(filePath).at("SceneInfo").array_items();235
}236
237
// frame = frame + 1; //index on the array start at 0 frames starts at 1238
239
}240
241
int mSEIBytesToRead = 509;242
-243
if(metadata)244
{245
delete(metadata);246
247
metadata = new uint8_t[mSEIBytesToRead];248
mPimpl->mCurrentStreamBit = 8;249
mPimpl->mCurrentStreamByte = 1;250
+ memset(metadata, 0, mSEIBytesToRead);251
252
- for(int j = 0; j < mSEIBytesToRead; ++j)253
- {254
- (metadata)[j] = 0;255
- }256
-257
- fillMetadataArray(fileData, frame, metadata);258
+ fillMetadataArray(fileData, frame, jsonType, metadata);259
mPimpl->setPayloadSize(metadata, 0, mPimpl->mCurrentStreamByte);260
return true;261
}262
263
{264
std::string path(filePath);265
JsonArray fileData = JsonHelper::readJsonArray(path);266
+ JsonType jsonType = LEGACY;267
if (fileData.empty())268
{269
- return -1;270
+ jsonType = LLC;271
+ fileData = JsonHelper::readJson(filePath).at("SceneInfo").array_items();272
}273
274
int numFrames = static_cast<int>(fileData.size());275
276
for (int frame = 0; frame < numFrames; ++frame)277
{278
metadata[frame] = new uint8_t[509];279
- for (int i = 0; i < 509; ++i)280
- {281
- metadata[frame][i] = 0;282
- }283
+ memset(metadata[frame], 0, 509);284
mPimpl->mCurrentStreamBit = 8;285
mPimpl->mCurrentStreamByte = 1;286
287
- fillMetadataArray(fileData, frame, metadata[frame]);288
-289
+ fillMetadataArray(fileData, frame, jsonType, metadata[frame]);290
mPimpl->setPayloadSize(metadata[frame], 0, mPimpl->mCurrentStreamByte);291
-292
}293
294
return numFrames;295
296
/* NOTE: We leave TWO BYTES of space for the payload */297
mPimpl->mCurrentStreamByte += 2;298
299
- fillMetadataArray(fileData, frame, metadata);300
+ fillMetadataArray(fileData, frame, LEGACY, metadata);301
302
/* Set payload in bytes 2 & 3 as indicated in Extended InfoFrame Type syntax */303
metadata[2] = (mPimpl->mCurrentStreamByte & 0xFF00) >> 8;304
305
306
int metadataFromJson::movieExtendedInfoFrameMetadataFromJson(const char* filePath, uint8_t **&metadata)307
{308
- std::string path(filePath);309
+ std::string path(filePath);310
JsonArray fileData = JsonHelper::readJsonArray(path);311
if(fileData.empty())312
{313
314
{315
metadata[frame] = new uint8_t[509];316
for(int i = 0; i < 509; ++i) 317
- {318
- metadata[frame][i] = 0;319
- }320
+ {321
+ metadata[frame][i] = 0;322
+ }323
mPimpl->mCurrentStreamBit = 8;324
mPimpl->mCurrentStreamByte = 0;325
326
327
/* NOTE: We leave TWO BYTES of space for the payload */328
mPimpl->mCurrentStreamByte += 2;329
330
- fillMetadataArray(fileData, frame, metadata[frame]);331
+ fillMetadataArray(fileData, frame, LEGACY, metadata[frame]);332
333
/* Set payload in bytes 2 & 3 as indicated in Extended InfoFrame Type syntax */334
metadata[frame][2] = (mPimpl->mCurrentStreamByte & 0xFF00) >> 8;335
336
return numFrames;337
}338
339
-void metadataFromJson::fillMetadataArray(const JsonArray &fileData, int frame, uint8_t *&metadata)340
+void metadataFromJson::fillMetadataArray(const JsonArray &fileData, int frame, const JsonType jsonType, uint8_t *&metadata)341
{342
const uint8_t countryCode = 0xB5;343
const uint16_t terminalProviderCode = 0x003C;344
345
mPimpl->appendBits(metadata, applicationIdentifier, 8);346
mPimpl->appendBits(metadata, applicationVersion, 8);347
348
- //Note: Validated only add up to two local selections, ignore the rest349
- JsonArray jsonArray = fileData[frame][JsonDataKeys::LocalParameters].array_items();350
- int ellipsesNum = static_cast<int>(jsonArray.size() > 2 ? 2 : jsonArray.size());351
- uint16_t numWindows = (uint16_t)fileData[frame][JsonDataKeys::NumberOfWindows].int_value();352
- mPimpl->appendBits(metadata, numWindows, 2);353
- for (int i = 0; i < ellipsesNum; ++i)354
+ uint16_t numWindows = 0;355
+ /* HDR10+ LLC doesn't consider local windows */356
+ if(jsonType & LLC)357
+ {358
+ numWindows = 1;359
+ mPimpl->appendBits(metadata, numWindows, 2);360
+ }361
+ else362
{363
- mPimpl->appendBits(metadata, jsonArray[i][EllipseSelectionNames::WindowData]364
- [EllipseSelectionNames::WindowUpperLeftCornerX].int_value(), 16);365
- mPimpl->appendBits(metadata, jsonArray[i][EllipseSelectionNames::WindowData]366
- [EllipseSelectionNames::WindowUpperLeftCornerY].int_value(), 16);367
- mPimpl->appendBits(metadata, jsonArray[i][EllipseSelectionNames::WindowData]368
- [EllipseSelectionNames::WindowLowerRightCornerX].int_value(), 16);369
- mPimpl->appendBits(metadata, jsonArray[i][EllipseSelectionNames::WindowData]370
- [EllipseSelectionNames::WindowLowerRightCornerY].int_value(), 16);371
+ //Note: Validated only add up to two local selections, ignore the rest372
+ JsonArray jsonArray = fileData[frame][JsonDataKeys::LocalParameters].array_items();373
+ int ellipsesNum = static_cast<int>(jsonArray.size() > 2 ? 2 : jsonArray.size());374
+ numWindows = (uint16_t)fileData[frame][JsonDataKeys::NumberOfWindows].int_value();375
+ mPimpl->appendBits(metadata, numWindows, 2);376
+ for (int i = 0; i < ellipsesNum; ++i)377
+ {378
+ mPimpl->appendBits(metadata, jsonArray[i][EllipseSelectionNames::WindowData]379
+ [EllipseSelectionNames::WindowUpperLeftCornerX].int_value(), 16);380
+ mPimpl->appendBits(metadata, jsonArray[i][EllipseSelectionNames::WindowData]381
+ [EllipseSelectionNames::WindowUpperLeftCornerY].int_value(), 16);382
+ mPimpl->appendBits(metadata, jsonArray[i][EllipseSelectionNames::WindowData]383
+ [EllipseSelectionNames::WindowLowerRightCornerX].int_value(), 16);384
+ mPimpl->appendBits(metadata, jsonArray[i][EllipseSelectionNames::WindowData]385
+ [EllipseSelectionNames::WindowLowerRightCornerY].int_value(), 16);386
387
- JsonObject ellipseJsonObject = jsonArray[i][EllipseNames::TagName].object_items();388
+ JsonObject ellipseJsonObject = jsonArray[i][EllipseNames::TagName].object_items();389
390
- mPimpl->appendBits(metadata,391
- static_cast<uint16_t>(ellipseJsonObject[EllipseNames::CenterOfEllipseX].int_value()),392
- 16);393
+ mPimpl->appendBits(metadata,394
+ static_cast<uint16_t>(ellipseJsonObject[EllipseNames::CenterOfEllipseX].int_value()),395
+ 16);396
397
- mPimpl->appendBits(metadata,398
- static_cast<uint16_t>(ellipseJsonObject[EllipseNames::CenterOfEllipseY].int_value()),399
- 16);400
+ mPimpl->appendBits(metadata,401
+ static_cast<uint16_t>(ellipseJsonObject[EllipseNames::CenterOfEllipseY].int_value()),402
+ 16);403
404
- int angle = ellipseJsonObject[EllipseNames::RotationAngle].int_value();405
- uint8_t rotationAngle = static_cast<uint8_t>((angle > 180.0) ? angle - 180.0 : angle);406
- mPimpl->appendBits(metadata, rotationAngle, 8);407
+ int angle = ellipseJsonObject[EllipseNames::RotationAngle].int_value();408
+ uint8_t rotationAngle = static_cast<uint8_t>((angle > 180.0) ? angle - 180.0 : angle);409
+ mPimpl->appendBits(metadata, rotationAngle, 8);410
411
- uint16_t semimajorExternalAxis =412
- static_cast<uint16_t>(ellipseJsonObject[EllipseNames::SemiMajorAxisExternalEllipse].int_value());413
+ uint16_t semimajorExternalAxis =414
+ static_cast<uint16_t>(ellipseJsonObject[EllipseNames::SemiMajorAxisExternalEllipse].int_value());415
416
- uint16_t semiminorExternalAxis =417
- static_cast<uint16_t>(ellipseJsonObject[EllipseNames::SemiMinorAxisExternalEllipse].int_value());418
+ uint16_t semiminorExternalAxis =419
+ static_cast<uint16_t>(ellipseJsonObject[EllipseNames::SemiMinorAxisExternalEllipse].int_value());420
421
- uint16_t semimajorInternalEllipse =422
- static_cast<uint16_t>(ellipseJsonObject[EllipseNames::SemiMajorAxisInternalEllipse].int_value());423
+ uint16_t semimajorInternalEllipse =424
+ static_cast<uint16_t>(ellipseJsonObject[EllipseNames::SemiMajorAxisInternalEllipse].int_value());425
426
- mPimpl->appendBits(metadata, semimajorInternalEllipse, 16);427
+ mPimpl->appendBits(metadata, semimajorInternalEllipse, 16);428
429
- mPimpl->appendBits(metadata, semimajorExternalAxis, 16);430
- mPimpl->appendBits(metadata, semiminorExternalAxis, 16);431
- uint8_t overlapProcessOption = static_cast<uint8_t>(ellipseJsonObject[EllipseNames::OverlapProcessOption].int_value());432
- //TODO: Uses Layering method, the value is "1"433
- mPimpl->appendBits(metadata, overlapProcessOption, 1);434
+ mPimpl->appendBits(metadata, semimajorExternalAxis, 16);435
+ mPimpl->appendBits(metadata, semiminorExternalAxis, 16);436
+ uint8_t overlapProcessOption = static_cast<uint8_t>(ellipseJsonObject[EllipseNames::OverlapProcessOption].int_value());437
+ //TODO: Uses Layering method, the value is "1"438
+ mPimpl->appendBits(metadata, overlapProcessOption, 1);439
+ }440
}441
+442
/* Targeted System Display Data */443
- uint32_t monitorPeak = fileData[frame][JsonDataKeys::TargetDisplayLuminance].int_value(); //500;444
+ uint32_t monitorPeak = fileData[frame][JsonDataKeys::TargetDisplayLuminance].int_value();445
mPimpl->appendBits(metadata, monitorPeak, 27);446
- //NOTE: Set as false for now, as requested447
+448
uint8_t targetedSystemDisplayActualPeakLuminanceFlag = 0;449
mPimpl->appendBits(metadata, targetedSystemDisplayActualPeakLuminanceFlag, 1);450
if (targetedSystemDisplayActualPeakLuminanceFlag)451
452
//TODO453
}454
455
- /* Max rgb values (maxScl)*/456
+ /* Max RGB values (maxScl)*/457
/* Luminance values/percentile for each window */458
for (int w = 0; w < numWindows; ++w)459
{460
Json lumObj = fileData[frame][LuminanceNames::TagName];461
LuminanceParameters luminanceData;462
- if (!mPimpl->luminanceParamFromJson(lumObj, luminanceData))463
+ if(!mPimpl->luminanceParamFromJson(lumObj, luminanceData, jsonType))464
{465
std::cout << "error parsing luminance parameters frame: " << w << std::endl;466
}467
468
- /* NOTE: Maxscl from 0 t 100,000 based on data that says in values of 0.00001469
+ /* NOTE: Maxscl from 0 to 100,000 based on data that says in values of 0.00001470
* one for each channel R,G,B471
*/472
-473
mPimpl->appendBits(metadata, static_cast<uint8_t>(((int)luminanceData.maxRLuminance & 0x10000) >> 16), 1);474
mPimpl->appendBits(metadata, static_cast<uint16_t>((int)luminanceData.maxRLuminance & 0xFFFF), 16);475
mPimpl->appendBits(metadata, static_cast<uint8_t>(((int)luminanceData.maxGLuminance & 0x10000) >> 16), 1);476
477
uint8_t numDistributionMaxrgbPercentiles = static_cast<uint8_t>(luminanceData.order);478
mPimpl->appendBits(metadata, numDistributionMaxrgbPercentiles, 4);479
480
- std::vector<unsigned int>percentilPercentages;481
- mPimpl->percentagesFromJson(lumObj, percentilPercentages);482
+ std::vector<unsigned int>percentilePercentages;483
+ mPimpl->percentagesFromJson(lumObj, percentilePercentages, jsonType);484
+485
for (int i = 0; i < numDistributionMaxrgbPercentiles; ++i)486
{487
- uint8_t distributionMaxrgbPercentage = static_cast<uint8_t>(percentilPercentages.at(i));488
+ uint8_t distributionMaxrgbPercentage = static_cast<uint8_t>(percentilePercentages.at(i));489
mPimpl->appendBits(metadata, distributionMaxrgbPercentage, 7);490
491
/* 17bits: 1bit then 16 */492
493
}494
495
/* 10bits: Fraction bright pixels */496
- uint16_t fractionBrightPixels = 1;497
+ uint16_t fractionBrightPixels = 0;498
mPimpl->appendBits(metadata, fractionBrightPixels, 10);499
500
}501
502
/* Bezier Curve Data */503
for (int w = 0; w < numWindows; ++w)504
{505
- uint8_t toneMappingFlag = 1;506
+ uint8_t toneMappingFlag = 0;507
/* Check if the window contains tone mapping bezier curve data and set toneMappingFlag appropriately */508
- //Json bezierData = fileData[frame][BezierCurveNames::TagName];509
BezierCurveData curveData;510
/* Select curve data based on global window */511
if (w == 0)512
- {513
- if (!mPimpl->bezierCurveFromJson(fileData[frame][BezierCurveNames::TagName], curveData))514
+ { 515
+ if (mPimpl->bezierCurveFromJson(fileData[frame][BezierCurveNames::TagName], curveData, jsonType))516
{517
- toneMappingFlag = 0;518
+ toneMappingFlag = 1;519
}520
}521
- /* Select curve data based on local window */522
+ /* Select curve data based on local window */523
else524
{525
- if (!mPimpl->bezierCurveFromJson(jsonArray[w - 1][BezierCurveNames::TagName], curveData))526
+ JsonArray jsonArray = fileData[frame][JsonDataKeys::LocalParameters].array_items();527
+ if (mPimpl->bezierCurveFromJson(jsonArray[w - 1][BezierCurveNames::TagName], curveData, jsonType))528
{529
- toneMappingFlag = 0;530
+ toneMappingFlag = 1;531
}532
} 533
mPimpl->appendBits(metadata, toneMappingFlag, 1);534
x265_2.7.tar.gz/source/dynamicHDR10/metadataFromJson.h -> x265_2.9.tar.gz/source/dynamicHDR10/metadataFromJson.h
Changed
31
1
2
#define METADATAFROMJSON_H3
4
#include<stdint.h>5
-#include "string"6
+#include<cstring>7
#include "JsonHelper.h"8
9
class metadataFromJson10
11
metadataFromJson();12
~metadataFromJson();13
14
+ enum JsonType{15
+ LEGACY,16
+ LLC17
+ };18
+ 19
20
/**21
* @brief frameMetadataFromJson: Generates a sigle frame metadata array from Json file with all22
23
24
class DynamicMetaIO;25
DynamicMetaIO *mPimpl;26
- void fillMetadataArray(const JsonArray &fileData, int frame, uint8_t *&metadata);27
+ void fillMetadataArray(const JsonArray &fileData, int frame, const JsonType jsonType, uint8_t *&metadata);28
};29
30
#endif // METADATAFROMJSON_H31
x265_2.7.tar.gz/source/encoder/analysis.cpp -> x265_2.9.tar.gz/source/encoder/analysis.cpp
Changed
634
1
2
using namespace X265_NS;3
4
/* An explanation of rate distortion levels (--rd-level)5
- * 6
+ *7
* rd-level 0 generates no recon per CU (NO RDO or Quant)8
*9
* sa8d selection between merge / skip / inter / intra and split10
11
for (uint32_t i = 0; i < cuGeom.numPartitions; i++)12
ctu.m_log2CUSize[i] = (uint8_t)m_param->maxLog2CUSize - ctu.m_cuDepth[i];13
}14
- if (m_param->analysisMultiPassRefine && m_param->rc.bStatRead)15
+ if (m_param->analysisMultiPassRefine && m_param->rc.bStatRead && (m_slice->m_sliceType != I_SLICE))16
{17
- m_multipassAnalysis = (analysis2PassFrameData*)m_frame->m_analysis2Pass.analysisFramedata;18
- m_multipassDepth = &m_multipassAnalysis->depth[ctu.m_cuAddr * ctu.m_numPartitions];19
- if (m_slice->m_sliceType != I_SLICE)20
+ int numPredDir = m_slice->isInterP() ? 1 : 2;21
+ m_reuseInterDataCTU = m_frame->m_analysisData.interData;22
+ for (int dir = 0; dir < numPredDir; dir++)23
{24
- int numPredDir = m_slice->isInterP() ? 1 : 2;25
- for (int dir = 0; dir < numPredDir; dir++)26
- {27
- m_multipassMv[dir] = &m_multipassAnalysis->m_mv[dir][ctu.m_cuAddr * ctu.m_numPartitions];28
- m_multipassMvpIdx[dir] = &m_multipassAnalysis->mvpIdx[dir][ctu.m_cuAddr * ctu.m_numPartitions];29
- m_multipassRef[dir] = &m_multipassAnalysis->ref[dir][ctu.m_cuAddr * ctu.m_numPartitions];30
- }31
- m_multipassModes = &m_multipassAnalysis->modes[ctu.m_cuAddr * ctu.m_numPartitions];32
+ m_reuseMv[dir] = &m_reuseInterDataCTU->mv[dir][ctu.m_cuAddr * ctu.m_numPartitions];33
+ m_reuseMvpIdx[dir] = &m_reuseInterDataCTU->mvpIdx[dir][ctu.m_cuAddr * ctu.m_numPartitions];34
}35
+ m_reuseRef = &m_reuseInterDataCTU->ref[ctu.m_cuAddr * ctu.m_numPartitions];36
+ m_reuseModes = &m_reuseInterDataCTU->modes[ctu.m_cuAddr * ctu.m_numPartitions];37
+ m_reuseDepth = &m_reuseInterDataCTU->depth[ctu.m_cuAddr * ctu.m_numPartitions];38
}39
-40
+ 41
if ((m_param->analysisSave || m_param->analysisLoad) && m_slice->m_sliceType != I_SLICE && m_param->analysisReuseLevel > 1 && m_param->analysisReuseLevel < 10)42
{43
int numPredDir = m_slice->isInterP() ? 1 : 2;44
- m_reuseInterDataCTU = (analysis_inter_data*)m_frame->m_analysisData.interData;45
+ m_reuseInterDataCTU = m_frame->m_analysisData.interData;46
m_reuseRef = &m_reuseInterDataCTU->ref [ctu.m_cuAddr * X265_MAX_PRED_MODE_PER_CTU * numPredDir];47
m_reuseDepth = &m_reuseInterDataCTU->depth[ctu.m_cuAddr * ctu.m_numPartitions];48
m_reuseModes = &m_reuseInterDataCTU->modes[ctu.m_cuAddr * ctu.m_numPartitions];49
50
51
if (m_slice->m_sliceType == I_SLICE)52
{53
- analysis_intra_data* intraDataCTU = (analysis_intra_data*)m_frame->m_analysisData.intraData;54
+ x265_analysis_intra_data* intraDataCTU = m_frame->m_analysisData.intraData;55
if (m_param->analysisLoad && m_param->analysisReuseLevel > 1)56
{57
memcpy(ctu.m_cuDepth, &intraDataCTU->depth[ctu.m_cuAddr * numPartition], sizeof(uint8_t) * numPartition);58
59
60
if (bCopyAnalysis)61
{62
- analysis_inter_data* interDataCTU = (analysis_inter_data*)m_frame->m_analysisData.interData;63
+ x265_analysis_inter_data* interDataCTU = m_frame->m_analysisData.interData;64
int posCTU = ctu.m_cuAddr * numPartition;65
memcpy(ctu.m_cuDepth, &interDataCTU->depth[posCTU], sizeof(uint8_t) * numPartition);66
memcpy(ctu.m_predMode, &interDataCTU->modes[posCTU], sizeof(uint8_t) * numPartition);67
68
69
if ((m_slice->m_sliceType == P_SLICE || m_param->bIntraInBFrames) && !m_param->bMVType)70
{71
- analysis_intra_data* intraDataCTU = (analysis_intra_data*)m_frame->m_analysisData.intraData;72
+ x265_analysis_intra_data* intraDataCTU = m_frame->m_analysisData.intraData;73
memcpy(ctu.m_lumaIntraDir, &intraDataCTU->modes[posCTU], sizeof(uint8_t) * numPartition);74
memcpy(ctu.m_chromaIntraDir, &intraDataCTU->chromaModes[posCTU], sizeof(uint8_t) * numPartition);75
}76
77
}78
else if ((m_param->analysisLoad && m_param->analysisReuseLevel == 10) || ((m_param->bMVType == AVC_INFO) && m_param->analysisReuseLevel >= 7 && ctu.m_numPartitions <= 16))79
{80
- analysis_inter_data* interDataCTU = (analysis_inter_data*)m_frame->m_analysisData.interData;81
+ x265_analysis_inter_data* interDataCTU = m_frame->m_analysisData.interData;82
int posCTU = ctu.m_cuAddr * numPartition;83
memcpy(ctu.m_cuDepth, &interDataCTU->depth[posCTU], sizeof(uint8_t) * numPartition);84
memcpy(ctu.m_predMode, &interDataCTU->modes[posCTU], sizeof(uint8_t) * numPartition);85
memcpy(ctu.m_partSize, &interDataCTU->partSize[posCTU], sizeof(uint8_t) * numPartition);86
if ((m_slice->m_sliceType == P_SLICE || m_param->bIntraInBFrames) && !(m_param->bMVType == AVC_INFO))87
{88
- analysis_intra_data* intraDataCTU = (analysis_intra_data*)m_frame->m_analysisData.intraData;89
+ x265_analysis_intra_data* intraDataCTU = m_frame->m_analysisData.intraData;90
memcpy(ctu.m_lumaIntraDir, &intraDataCTU->modes[posCTU], sizeof(uint8_t) * numPartition);91
memcpy(ctu.m_chromaIntraDir, &intraDataCTU->chromaModes[posCTU], sizeof(uint8_t) * numPartition);92
}93
94
bool mightSplit = !(cuGeom.flags & CUGeom::LEAF);95
bool mightNotSplit = !(cuGeom.flags & CUGeom::SPLIT_MANDATORY);96
97
- bool bAlreadyDecided = parentCTU.m_lumaIntraDir[cuGeom.absPartIdx] != (uint8_t)ALL_IDX;98
- bool bDecidedDepth = parentCTU.m_cuDepth[cuGeom.absPartIdx] == depth;99
+ bool bAlreadyDecided = m_param->intraRefine != 4 && parentCTU.m_lumaIntraDir[cuGeom.absPartIdx] != (uint8_t)ALL_IDX;100
+ bool bDecidedDepth = m_param->intraRefine != 4 && parentCTU.m_cuDepth[cuGeom.absPartIdx] == depth;101
int split = 0;102
- if (m_param->intraRefine)103
+ if (m_param->intraRefine && m_param->intraRefine != 4)104
{105
- split = ((cuGeom.log2CUSize == (uint32_t)(g_log2Size[m_param->minCUSize] + 1)) && bDecidedDepth);106
+ split = m_param->scaleFactor && bDecidedDepth && (!mightNotSplit || 107
+ ((cuGeom.log2CUSize == (uint32_t)(g_log2Size[m_param->minCUSize] + 1))));108
if (cuGeom.log2CUSize == (uint32_t)(g_log2Size[m_param->minCUSize]) && !bDecidedDepth)109
bAlreadyDecided = false;110
}111
112
if (bAlreadyDecided)113
{114
- if (bDecidedDepth)115
+ if (bDecidedDepth && mightNotSplit)116
{117
Mode& mode = md.pred[0];118
md.bestMode = &mode;119
120
121
if (m_evaluateInter)122
{123
- if (m_param->interRefine == 2)124
+ if (m_refineLevel == 2)125
{126
if (parentCTU.m_predMode[cuGeom.absPartIdx] == MODE_SKIP)127
skipModes = true;128
129
}130
}131
}132
- if (m_param->analysisMultiPassRefine && m_param->rc.bStatRead && m_multipassAnalysis)133
+ if (m_param->analysisMultiPassRefine && m_param->rc.bStatRead && m_reuseInterDataCTU)134
{135
- if (mightNotSplit && depth == m_multipassDepth[cuGeom.absPartIdx])136
+ if (mightNotSplit && depth == m_reuseDepth[cuGeom.absPartIdx])137
{138
- if (m_multipassModes[cuGeom.absPartIdx] == MODE_SKIP)139
+ if (m_reuseModes[cuGeom.absPartIdx] == MODE_SKIP)140
{141
md.pred[PRED_MERGE].cu.initSubCU(parentCTU, cuGeom, qp);142
md.pred[PRED_SKIP].cu.initSubCU(parentCTU, cuGeom, qp);143
144
md.pred[PRED_SKIP].cu.initSubCU(parentCTU, cuGeom, qp);145
checkMerge2Nx2N_rd0_4(md.pred[PRED_SKIP], md.pred[PRED_MERGE], cuGeom);146
if (m_param->rdLevel)147
- skipModes = (m_param->bEnableEarlySkip || m_param->interRefine == 2)148
+ skipModes = (m_param->bEnableEarlySkip || m_refineLevel == 2)149
&& md.bestMode && md.bestMode->cu.isSkipped(0); // TODO: sa8d threshold per depth150
}151
if (md.bestMode && m_param->bEnableRecursionSkip && !bCtuInfoCheck && !(m_param->bMVType && m_param->analysisReuseLevel == 7 && (m_modeFlag[0] || m_modeFlag[1])))152
153
154
if (m_evaluateInter)155
{156
- if (m_param->interRefine == 2)157
+ if (m_refineLevel == 2)158
{159
if (parentCTU.m_predMode[cuGeom.absPartIdx] == MODE_SKIP)160
skipModes = true;161
162
}163
}164
165
- if (m_param->analysisMultiPassRefine && m_param->rc.bStatRead && m_multipassAnalysis)166
+ if (m_param->analysisMultiPassRefine && m_param->rc.bStatRead && m_reuseInterDataCTU)167
{168
- if (mightNotSplit && depth == m_multipassDepth[cuGeom.absPartIdx])169
+ if (mightNotSplit && depth == m_reuseDepth[cuGeom.absPartIdx])170
{171
- if (m_multipassModes[cuGeom.absPartIdx] == MODE_SKIP)172
+ if (m_reuseModes[cuGeom.absPartIdx] == MODE_SKIP)173
{174
md.pred[PRED_MERGE].cu.initSubCU(parentCTU, cuGeom, qp);175
md.pred[PRED_SKIP].cu.initSubCU(parentCTU, cuGeom, qp);176
177
md.pred[PRED_SKIP].cu.initSubCU(parentCTU, cuGeom, qp);178
md.pred[PRED_MERGE].cu.initSubCU(parentCTU, cuGeom, qp);179
checkMerge2Nx2N_rd5_6(md.pred[PRED_SKIP], md.pred[PRED_MERGE], cuGeom);180
- skipModes = (m_param->bEnableEarlySkip || m_param->interRefine == 2) &&181
+ skipModes = (m_param->bEnableEarlySkip || m_refineLevel == 2) &&182
md.bestMode && !md.bestMode->cu.getQtRootCbf(0);183
refMasks[0] = allSplitRefs;184
md.pred[PRED_2Nx2N].cu.initSubCU(parentCTU, cuGeom, qp);185
186
bool mightNotSplit = !(cuGeom.flags & CUGeom::SPLIT_MANDATORY);187
bool bDecidedDepth = parentCTU.m_cuDepth[cuGeom.absPartIdx] == depth;188
189
- int split = (m_param->interRefine && cuGeom.log2CUSize == (uint32_t)(g_log2Size[m_param->minCUSize] + 1) && bDecidedDepth);190
+ TrainingData td;191
+ td.init(parentCTU, cuGeom);192
193
- if (bDecidedDepth)194
+ if (!m_param->bDynamicRefine)195
+ m_refineLevel = m_param->interRefine;196
+ else197
+ m_refineLevel = m_frame->m_classifyFrame ? 1 : 3;198
+ int split = (m_param->scaleFactor && bDecidedDepth && (!mightNotSplit || 199
+ (m_refineLevel && cuGeom.log2CUSize == (uint32_t)(g_log2Size[m_param->minCUSize] + 1))));200
+ td.split = split;201
+202
+ if (bDecidedDepth && mightNotSplit)203
{204
setLambdaFromQP(parentCTU, qp, lqp);205
206
207
md.bestMode = &mode;208
mode.cu.initSubCU(parentCTU, cuGeom, qp);209
PartSize size = (PartSize)parentCTU.m_partSize[cuGeom.absPartIdx];210
- if (parentCTU.isIntra(cuGeom.absPartIdx) && m_param->interRefine < 2)211
+ if (parentCTU.isIntra(cuGeom.absPartIdx) && m_refineLevel < 2)212
{213
- bool reuseModes = !((m_param->intraRefine == 3) ||214
- (m_param->intraRefine == 2 && parentCTU.m_lumaIntraDir[cuGeom.absPartIdx] > DC_IDX));215
- if (reuseModes)216
+ if (m_param->intraRefine == 4)217
+ compressIntraCU(parentCTU, cuGeom, qp);218
+ else219
{220
- memcpy(mode.cu.m_lumaIntraDir, parentCTU.m_lumaIntraDir + cuGeom.absPartIdx, cuGeom.numPartitions);221
- memcpy(mode.cu.m_chromaIntraDir, parentCTU.m_chromaIntraDir + cuGeom.absPartIdx, cuGeom.numPartitions);222
+ bool reuseModes = !((m_param->intraRefine == 3) ||223
+ (m_param->intraRefine == 2 && parentCTU.m_lumaIntraDir[cuGeom.absPartIdx] > DC_IDX));224
+ if (reuseModes)225
+ {226
+ memcpy(mode.cu.m_lumaIntraDir, parentCTU.m_lumaIntraDir + cuGeom.absPartIdx, cuGeom.numPartitions);227
+ memcpy(mode.cu.m_chromaIntraDir, parentCTU.m_chromaIntraDir + cuGeom.absPartIdx, cuGeom.numPartitions);228
+ }229
+ checkIntra(mode, cuGeom, size);230
}231
- checkIntra(mode, cuGeom, size);232
}233
- else if (!parentCTU.isIntra(cuGeom.absPartIdx) && m_param->interRefine < 2)234
+ else if (!parentCTU.isIntra(cuGeom.absPartIdx) && m_refineLevel < 2)235
{236
mode.cu.copyFromPic(parentCTU, cuGeom, m_csp, false);237
uint32_t numPU = parentCTU.getNumPartInter(cuGeom.absPartIdx);238
for (uint32_t part = 0; part < numPU; part++)239
{240
PredictionUnit pu(mode.cu, cuGeom, part);241
- if (m_param->analysisReuseLevel >= 7)242
+ if ((m_param->analysisLoad && m_param->analysisReuseLevel == 10) || (m_param->bMVType == AVC_INFO && m_param->analysisReuseLevel >= 7))243
{244
- analysis_inter_data* interDataCTU = (analysis_inter_data*)m_frame->m_analysisData.interData;245
+ x265_analysis_inter_data* interDataCTU = m_frame->m_analysisData.interData;246
int cuIdx = (mode.cu.m_cuAddr * parentCTU.m_numPartitions) + cuGeom.absPartIdx;247
mode.cu.m_mergeFlag[pu.puAbsPartIdx] = interDataCTU->mergeFlag[cuIdx + part];248
mode.cu.setPUInterDir(interDataCTU->interDir[cuIdx + part], pu.puAbsPartIdx, part);249
for (int list = 0; list < m_slice->isInterB() + 1; list++)250
{251
- mode.cu.setPUMv(list, interDataCTU->mv[list][cuIdx + part], pu.puAbsPartIdx, part);252
+ mode.cu.setPUMv(list, interDataCTU->mv[list][cuIdx + part].word, pu.puAbsPartIdx, part);253
mode.cu.setPURefIdx(list, interDataCTU->refIdx[list][cuIdx + part], pu.puAbsPartIdx, part);254
mode.cu.m_mvpIdx[list][pu.puAbsPartIdx] = interDataCTU->mvpIdx[list][cuIdx + part];255
}256
if (!mode.cu.m_mergeFlag[pu.puAbsPartIdx])257
{258
- if (m_param->mvRefine)259
+ if (m_param->mvRefine || m_param->interRefine == 1)260
m_me.setSourcePU(*mode.fencYuv, pu.ctuAddr, pu.cuAbsPartIdx, pu.puAbsPartIdx, pu.width, pu.height, m_param->searchMethod, m_param->subpelRefine, false);261
//AMVP262
MV mvc[(MD_ABOVE_LEFT + 1) * 2 + 2];263
264
int ref = mode.cu.m_refIdx[list][pu.puAbsPartIdx];265
if (ref == -1)266
continue;267
- mode.cu.getPMV(mode.interNeighbours, list, ref, mode.amvpCand[list][ref], mvc);268
- MV mvp = mode.amvpCand[list][ref][mode.cu.m_mvpIdx[list][pu.puAbsPartIdx]];269
- if (m_param->mvRefine)270
+ MV mvp;271
+272
+ int numMvc = mode.cu.getPMV(mode.interNeighbours, list, ref, mode.amvpCand[list][ref], mvc);273
+ if (m_param->interRefine != 1)274
+ mvp = mode.amvpCand[list][ref][mode.cu.m_mvpIdx[list][pu.puAbsPartIdx]];275
+ else276
+ mvp = interDataCTU->mv[list][cuIdx + part].word;277
+ if (m_param->mvRefine || m_param->interRefine == 1)278
{279
MV outmv;280
- searchMV(mode, pu, list, ref, outmv);281
+ searchMV(mode, pu, list, ref, outmv, mvp, numMvc, mvc);282
mode.cu.setPUMv(list, outmv, pu.puAbsPartIdx, part);283
}284
- mode.cu.m_mvd[list][pu.puAbsPartIdx] = mode.cu.m_mv[list][pu.puAbsPartIdx] - mvp;285
+ mode.cu.m_mvd[list][pu.puAbsPartIdx] = mode.cu.m_mv[list][pu.puAbsPartIdx] - mode.amvpCand[list][ref][mode.cu.m_mvpIdx[list][pu.puAbsPartIdx]]/*mvp*/;286
}287
}288
- else if(m_param->scaleFactor)289
+ else290
{291
MVField candMvField[MRG_MAX_NUM_CANDS][2]; // double length for mv of both lists292
uint8_t candDir[MRG_MAX_NUM_CANDS];293
mode.cu.getInterMergeCandidates(pu.puAbsPartIdx, part, candMvField, candDir);294
uint8_t mvpIdx = mode.cu.m_mvpIdx[0][pu.puAbsPartIdx];295
+ if (mode.cu.isBipredRestriction())296
+ {297
+ /* do not allow bidir merge candidates if PU is smaller than 8x8, drop L1 reference */298
+ if (candDir[mvpIdx] == 3)299
+ {300
+ candDir[mvpIdx] = 1;301
+ candMvField[mvpIdx][1].refIdx = REF_NOT_VALID;302
+ }303
+ }304
mode.cu.setPUInterDir(candDir[mvpIdx], pu.puAbsPartIdx, part);305
mode.cu.setPUMv(0, candMvField[mvpIdx][0].mv, pu.puAbsPartIdx, part);306
mode.cu.setPUMv(1, candMvField[mvpIdx][1].mv, pu.puAbsPartIdx, part);307
308
}309
motionCompensation(mode.cu, pu, mode.predYuv, true, (m_csp != X265_CSP_I400 && m_frame->m_fencPic->m_picCsp != X265_CSP_I400));310
}311
- if (!m_param->interRefine && parentCTU.isSkipped(cuGeom.absPartIdx))312
+ if (!m_param->interRefine && !m_param->bDynamicRefine && parentCTU.isSkipped(cuGeom.absPartIdx))313
encodeResAndCalcRdSkipCU(mode);314
else315
encodeResAndCalcRdInterCU(mode, cuGeom);316
317
checkDQP(mode, cuGeom);318
}319
320
- if (m_param->interRefine < 2)321
+ if (m_refineLevel < 2)322
{323
if (m_bTryLossless)324
tryLossless(cuGeom);325
326
}327
}328
329
- if (m_param->interRefine > 1 || (m_param->interRefine && parentCTU.m_predMode[cuGeom.absPartIdx] == MODE_SKIP && !mode.cu.isSkipped(0)))330
+ if (m_param->bDynamicRefine)331
+ classifyCU(parentCTU,cuGeom, *md.bestMode, td);332
+333
+ if (m_refineLevel > 1 || (m_refineLevel && parentCTU.m_predMode[cuGeom.absPartIdx] == MODE_SKIP && !mode.cu.isSkipped(0)))334
{335
m_evaluateInter = 1;336
m_param->rdLevel > 4 ? compressInterCU_rd5_6(parentCTU, cuGeom, qp) : compressInterCU_rd0_4(parentCTU, cuGeom, qp);337
338
else339
updateModeCost(*splitPred);340
341
- if (m_param->interRefine)342
+ if (m_refineLevel)343
{344
if (m_param->rdLevel > 1)345
checkBestMode(*splitPred, cuGeom.depth);346
347
md.bestMode->cu.copyToPic(depth);348
md.bestMode->reconYuv.copyToPicYuv(*m_frame->m_reconPic, parentCTU.m_cuAddr, cuGeom.absPartIdx);349
}350
+ if (m_param->bDynamicRefine && bDecidedDepth)351
+ trainCU(parentCTU, cuGeom, *md.bestMode, td);352
+}353
+354
+void Analysis::classifyCU(const CUData& ctu, const CUGeom& cuGeom, const Mode& bestMode, TrainingData& trainData)355
+{356
+ uint32_t depth = cuGeom.depth;357
+ trainData.cuVariance = calculateCUVariance(ctu, cuGeom);358
+ if (m_frame->m_classifyFrame)359
+ {360
+ uint64_t diffRefine[X265_REFINE_INTER_LEVELS];361
+ uint64_t diffRefineRd[X265_REFINE_INTER_LEVELS];362
+ float probRefine[X265_REFINE_INTER_LEVELS] = { 0 };363
+ uint8_t varRefineLevel = 1;364
+ uint8_t rdRefineLevel = 1;365
+ uint64_t cuCost = bestMode.rdCost;366
+ int offset = (depth * X265_REFINE_INTER_LEVELS);367
+ if (cuCost < m_frame->m_classifyRd[offset])368
+ m_refineLevel = 1;369
+ else370
+ {371
+ uint64_t trainingCount = 0;372
+ for (uint8_t i = 0; i < X265_REFINE_INTER_LEVELS; i++)373
+ {374
+ offset = (depth * X265_REFINE_INTER_LEVELS) + i;375
+ trainingCount += m_frame->m_classifyCount[offset];376
+ }377
+ for (uint8_t i = 0; i < X265_REFINE_INTER_LEVELS; i++)378
+ {379
+ offset = (depth * X265_REFINE_INTER_LEVELS) + i;380
+ /* Calculate distance values */381
+ diffRefine[i] = abs((int64_t)(trainData.cuVariance - m_frame->m_classifyVariance[offset]));382
+ diffRefineRd[i] = abs((int64_t)(cuCost - m_frame->m_classifyRd[offset]));383
+384
+ /* Calculate prior probability - ranges between 0 and 1 */385
+ if (trainingCount)386
+ probRefine[i] = ((float)m_frame->m_classifyCount[offset] / (float)trainingCount);387
+388
+ /* Bayesian classification - P(c|x)P(x) = P(x|c)P(c)389
+ P(c|x) is the posterior probability of class given predictor.390
+ P(c) is the prior probability of class.391
+ P(x|c) is the likelihood which is the probability of predictor given class.392
+ P(x) is the prior probability of predictor.*/393
+ int curRefineLevel = m_refineLevel - 1;394
+ if ((diffRefine[i] * probRefine[curRefineLevel]) < (diffRefine[curRefineLevel] * probRefine[i]))395
+ varRefineLevel = i + 1;396
+ if ((diffRefineRd[i] * probRefine[curRefineLevel]) < (diffRefineRd[curRefineLevel] * probRefine[i]))397
+ rdRefineLevel = i + 1;398
+ }399
+ m_refineLevel = X265_MAX(varRefineLevel, rdRefineLevel);400
+ }401
+ }402
+}403
+404
+void Analysis::trainCU(const CUData& ctu, const CUGeom& cuGeom, const Mode& bestMode, TrainingData& trainData)405
+{406
+ uint32_t depth = cuGeom.depth;407
+ int classify = 1;408
+ if (!m_frame->m_classifyFrame)409
+ {410
+ /* classify = 1 : CUs for which the save data matches with that after encoding with refine-inter 3411
+ and CUs that has split.412
+ classify = 2 : CUs which are encoded as simple modes (Skip/Merge/2Nx2N).413
+ classify = 3 : CUs encoded as any other mode. */414
+415
+ bool refineInter0 = (trainData.predMode == ctu.m_predMode[cuGeom.absPartIdx] &&416
+ trainData.partSize == ctu.m_partSize[cuGeom.absPartIdx] &&417
+ trainData.mergeFlag == ctu.m_mergeFlag[cuGeom.absPartIdx]);418
+ bool refineInter1 = (depth == m_param->maxCUDepth - 1) && trainData.split;419
+ if (refineInter0 || refineInter1)420
+ classify = 1;421
+ else if (trainData.partSize == SIZE_2Nx2N && trainData.partSize == ctu.m_partSize[cuGeom.absPartIdx])422
+ classify = 2;423
+ else424
+ classify = 3;425
+ }426
+ else427
+ classify = m_refineLevel;428
+ uint64_t cuCost = bestMode.rdCost;429
+ int offset = (depth * X265_REFINE_INTER_LEVELS) + classify - 1;430
+ ctu.m_collectCURd[offset] += cuCost;431
+ ctu.m_collectCUVariance[offset] += trainData.cuVariance;432
+ ctu.m_collectCUCount[offset]++;433
}434
435
/* sets md.bestMode if a valid merge candidate is found, else leaves it NULL */436
437
}438
}439
440
- if (m_param->analysisMultiPassRefine && m_param->rc.bStatRead && m_multipassAnalysis)441
+ if (m_param->analysisMultiPassRefine && m_param->rc.bStatRead && m_reuseInterDataCTU)442
{443
uint32_t numPU = interMode.cu.getNumPartInter(0);444
for (uint32_t part = 0; part < numPU; part++)445
446
MotionData* bestME = interMode.bestME[part];447
for (int32_t i = 0; i < numPredDir; i++)448
{449
- bestME[i].ref = m_multipassRef[i][cuGeom.absPartIdx];450
- bestME[i].mv = m_multipassMv[i][cuGeom.absPartIdx];451
- bestME[i].mvpIdx = m_multipassMvpIdx[i][cuGeom.absPartIdx];452
+ int* ref = &m_reuseRef[i * m_frame->m_analysisData.numPartitions * m_frame->m_analysisData.numCUsInFrame];453
+ bestME[i].ref = ref[cuGeom.absPartIdx];454
+ bestME[i].mv = m_reuseMv[i][cuGeom.absPartIdx].word;455
+ bestME[i].mvpIdx = m_reuseMvpIdx[i][cuGeom.absPartIdx];456
}457
}458
}459
460
}461
}462
463
- if (m_param->analysisMultiPassRefine && m_param->rc.bStatRead && m_multipassAnalysis)464
+ if (m_param->analysisMultiPassRefine && m_param->rc.bStatRead && m_reuseInterDataCTU)465
{466
uint32_t numPU = interMode.cu.getNumPartInter(0);467
for (uint32_t part = 0; part < numPU; part++)468
469
MotionData* bestME = interMode.bestME[part];470
for (int32_t i = 0; i < numPredDir; i++)471
{472
- bestME[i].ref = m_multipassRef[i][cuGeom.absPartIdx];473
- bestME[i].mv = m_multipassMv[i][cuGeom.absPartIdx];474
- bestME[i].mvpIdx = m_multipassMvpIdx[i][cuGeom.absPartIdx];475
+ int* ref = &m_reuseRef[i * m_frame->m_analysisData.numPartitions * m_frame->m_analysisData.numCUsInFrame];476
+ bestME[i].ref = ref[cuGeom.absPartIdx];477
+ bestME[i].mv = m_reuseMv[i][cuGeom.absPartIdx].word;478
+ bestME[i].mvpIdx = m_reuseMvpIdx[i][cuGeom.absPartIdx];479
}480
}481
}482
483
pixel *fref0 = m_slice->m_mref[0][ref0].getLumaAddr(pu.ctuAddr, pu.cuAbsPartIdx);484
pixel *fref1 = m_slice->m_mref[1][ref1].getLumaAddr(pu.ctuAddr, pu.cuAbsPartIdx);485
intptr_t refStride = m_slice->m_mref[0][0].lumaStride;486
-487
- primitives.pu[partEnum].pixelavg_pp(tmpPredYuv.m_buf[0], tmpPredYuv.m_size, fref0, refStride, fref1, refStride, 32);488
+ primitives.pu[partEnum].pixelavg_pp[(tmpPredYuv.m_size % 64 == 0) && (refStride % 64 == 0)](tmpPredYuv.m_buf[0], tmpPredYuv.m_size, fref0, refStride, fref1, refStride, 32);489
zsa8d = primitives.cu[partEnum].sa8d(fencYuv.m_buf[0], fencYuv.m_size, tmpPredYuv.m_buf[0], tmpPredYuv.m_size);490
}491
-492
uint32_t bits0 = bestME[0].bits - m_me.bitcost(bestME[0].mv, mvp0) + m_me.bitcost(mvzero, mvp0);493
uint32_t bits1 = bestME[1].bits - m_me.bitcost(bestME[1].mv, mvp1) + m_me.bitcost(mvzero, mvp1);494
uint32_t zcost = zsa8d + m_rdCost.getCost(bits0) + m_rdCost.getCost(bits1);495
496
* resiYuv. Generate the recon pixels by adding it to the prediction */497
498
if (cu.m_cbf[0][0])499
- primitives.cu[sizeIdx].add_ps(reconPic.getLumaAddr(cu.m_cuAddr, absPartIdx), reconPic.m_stride,500
- predY, resiYuv.m_buf[0], predYuv.m_size, resiYuv.m_size);501
+ {502
+ bool reconPicAlign = (reconPic.m_cuOffsetY[cu.m_cuAddr] + reconPic.m_buOffsetY[absPartIdx]) % 64 == 0;503
+ bool predYalign = predYuv.getAddrOffset(absPartIdx, predYuv.m_size) % 64 == 0;504
+ primitives.cu[sizeIdx].add_ps[reconPicAlign && predYalign && (reconPic.m_stride % 64 == 0) && (predYuv.m_size % 64 == 0) &&505
+ (resiYuv.m_size % 64 == 0)](reconPic.getLumaAddr(cu.m_cuAddr, absPartIdx), reconPic.m_stride, predY, resiYuv.m_buf[0], predYuv.m_size, resiYuv.m_size);506
+ }507
else508
primitives.cu[sizeIdx].copy_pp(reconPic.getLumaAddr(cu.m_cuAddr, absPartIdx), reconPic.m_stride,509
predY, predYuv.m_size);510
511
{512
pixel* predU = predYuv.getCbAddr(absPartIdx);513
pixel* predV = predYuv.getCrAddr(absPartIdx);514
- if (cu.m_cbf[1][0])515
- primitives.chroma[m_csp].cu[sizeIdx].add_ps(reconPic.getCbAddr(cu.m_cuAddr, absPartIdx), reconPic.m_strideC,516
- predU, resiYuv.m_buf[1], predYuv.m_csize, resiYuv.m_csize);517
+ if (cu.m_cbf[1][0])518
+ {519
+ bool reconPicAlign = (reconPic.m_cuOffsetC[cu.m_cuAddr] + reconPic.m_buOffsetC[absPartIdx]) % 64 == 0;520
+ bool predUalign = predYuv.getChromaAddrOffset(absPartIdx) % 64 == 0;521
+ primitives.chroma[m_csp].cu[sizeIdx].add_ps[reconPicAlign && predUalign && (reconPic.m_strideC % 64 == 0) && (predYuv.m_csize % 64 == 0) &&522
+ (resiYuv.m_csize % 64 == 0)](reconPic.getCbAddr(cu.m_cuAddr, absPartIdx), reconPic.m_strideC, predU, resiYuv.m_buf[1], predYuv.m_csize, resiYuv.m_csize);523
+ }524
else525
primitives.chroma[m_csp].cu[sizeIdx].copy_pp(reconPic.getCbAddr(cu.m_cuAddr, absPartIdx), reconPic.m_strideC,526
predU, predYuv.m_csize);527
528
if (cu.m_cbf[2][0])529
- primitives.chroma[m_csp].cu[sizeIdx].add_ps(reconPic.getCrAddr(cu.m_cuAddr, absPartIdx), reconPic.m_strideC,530
- predV, resiYuv.m_buf[2], predYuv.m_csize, resiYuv.m_csize);531
+ {532
+ bool reconPicAlign = (reconPic.m_cuOffsetC[cu.m_cuAddr] + reconPic.m_buOffsetC[absPartIdx]) % 64 == 0;533
+ bool predValign = predYuv.getChromaAddrOffset(absPartIdx) % 64 == 0;534
+ primitives.chroma[m_csp].cu[sizeIdx].add_ps[reconPicAlign && predValign && (reconPic.m_strideC % 64 == 0) && (predYuv.m_csize % 64 == 0) &&535
+ (resiYuv.m_csize % 64 == 0)](reconPic.getCrAddr(cu.m_cuAddr, absPartIdx), reconPic.m_strideC, predV, resiYuv.m_buf[2], predYuv.m_csize, resiYuv.m_csize);536
+ }537
else538
primitives.chroma[m_csp].cu[sizeIdx].copy_pp(reconPic.getCrAddr(cu.m_cuAddr, absPartIdx), reconPic.m_strideC,539
predV, predYuv.m_csize);540
541
return false;542
}543
544
+uint32_t Analysis::calculateCUVariance(const CUData& ctu, const CUGeom& cuGeom)545
+{546
+ uint32_t cuVariance = 0;547
+ uint32_t *blockVariance = m_frame->m_lowres.blockVariance;548
+ int loopIncr = (m_param->rc.qgSize == 8) ? 8 : 16;549
+550
+ uint32_t width = m_frame->m_fencPic->m_picWidth;551
+ uint32_t height = m_frame->m_fencPic->m_picHeight;552
+ uint32_t block_x = ctu.m_cuPelX + g_zscanToPelX[cuGeom.absPartIdx];553
+ uint32_t block_y = ctu.m_cuPelY + g_zscanToPelY[cuGeom.absPartIdx];554
+ uint32_t maxCols = (m_frame->m_fencPic->m_picWidth + (loopIncr - 1)) / loopIncr;555
+ uint32_t blockSize = m_param->maxCUSize >> cuGeom.depth;556
+ uint32_t cnt = 0; 557
+558
+ for (uint32_t block_yy = block_y; block_yy < block_y + blockSize && block_yy < height; block_yy += loopIncr)559
+ {560
+ for (uint32_t block_xx = block_x; block_xx < block_x + blockSize && block_xx < width; block_xx += loopIncr)561
+ {562
+ uint32_t idx = ((block_yy / loopIncr) * (maxCols)) + (block_xx / loopIncr);563
+ cuVariance += blockVariance[idx];564
+ cnt++;565
+ }566
+ }567
+ 568
+ return cuVariance / cnt;569
+}570
+571
int Analysis::calculateQpforCuSize(const CUData& ctu, const CUGeom& cuGeom, int32_t complexCheck, double baseQp)572
{573
FrameData& curEncData = *m_frame->m_encData;574
575
576
if (m_param->analysisMultiPassDistortion && m_param->rc.bStatRead)577
{578
- m_multipassAnalysis = (analysis2PassFrameData*)m_frame->m_analysis2Pass.analysisFramedata;579
- if ((m_multipassAnalysis->threshold[ctu.m_cuAddr] < 0.9 || m_multipassAnalysis->threshold[ctu.m_cuAddr] > 1.1)580
- && m_multipassAnalysis->highDistortionCtuCount && m_multipassAnalysis->lowDistortionCtuCount)581
- qp += m_multipassAnalysis->offset[ctu.m_cuAddr];582
+ x265_analysis_distortion_data* distortionData = m_frame->m_analysisData.distortionData;583
+ if ((distortionData->threshold[ctu.m_cuAddr] < 0.9 || distortionData->threshold[ctu.m_cuAddr] > 1.1)584
+ && distortionData->highDistortionCtuCount && distortionData->lowDistortionCtuCount)585
+ qp += distortionData->offset[ctu.m_cuAddr];586
}587
588
- int loopIncr;589
- if (m_param->rc.qgSize == 8)590
- loopIncr = 8;591
- else592
- loopIncr = 16;593
+ int loopIncr = (m_param->rc.qgSize == 8) ? 8 : 16;594
+595
/* Use cuTree offsets if cuTree enabled and frame is referenced, else use AQ offsets */596
bool isReferenced = IS_REFERENCED(m_frame);597
- double *qpoffs;598
- if (complexCheck)599
- qpoffs = m_frame->m_lowres.qpAqOffset;600
- else601
- qpoffs = (isReferenced && m_param->rc.cuTree) ? m_frame->m_lowres.qpCuTreeOffset : m_frame->m_lowres.qpAqOffset;602
+ double *qpoffs = (isReferenced && m_param->rc.cuTree && !complexCheck) ? m_frame->m_lowres.qpCuTreeOffset :603
+ m_frame->m_lowres.qpAqOffset;604
if (qpoffs)605
{606
uint32_t width = m_frame->m_fencPic->m_picWidth;607
608
uint32_t blockSize = m_param->maxCUSize >> cuGeom.depth;609
double qp_offset = 0;610
uint32_t cnt = 0;611
- uint32_t idx;612
-613
for (uint32_t block_yy = block_y; block_yy < block_y + blockSize && block_yy < height; block_yy += loopIncr)614
{615
for (uint32_t block_xx = block_x; block_xx < block_x + blockSize && block_xx < width; block_xx += loopIncr)616
{617
- idx = ((block_yy / loopIncr) * (maxCols)) + (block_xx / loopIncr);618
+ uint32_t idx = ((block_yy / loopIncr) * (maxCols)) + (block_xx / loopIncr);619
qp_offset += qpoffs[idx];620
cnt++;621
}622
623
int32_t offset = (int32_t)(qp_offset * 100 + .5);624
double threshold = (1 - ((x265_ADAPT_RD_STRENGTH - m_param->dynamicRd) * 0.5));625
int32_t max_threshold = (int32_t)(threshold * 100 + .5);626
- if (offset < max_threshold)627
- return 1;628
- else629
- return 0;630
+ return (offset < max_threshold);631
}632
}633
634
x265_2.7.tar.gz/source/encoder/analysis.h -> x265_2.9.tar.gz/source/encoder/analysis.h
Changed
69
1
2
3
protected:4
/* Analysis data for save/load mode, writes/reads data based on absPartIdx */5
- analysis_inter_data* m_reuseInterDataCTU;6
- int32_t* m_reuseRef;7
- uint8_t* m_reuseDepth;8
- uint8_t* m_reuseModes;9
- uint8_t* m_reusePartSize;10
- uint8_t* m_reuseMergeFlag;11
+ x265_analysis_inter_data* m_reuseInterDataCTU;12
+ int32_t* m_reuseRef;13
+ uint8_t* m_reuseDepth;14
+ uint8_t* m_reuseModes;15
+ uint8_t* m_reusePartSize;16
+ uint8_t* m_reuseMergeFlag;17
+ x265_analysis_MV* m_reuseMv[2];18
+ uint8_t* m_reuseMvpIdx[2];19
20
uint32_t m_splitRefIdx[4];21
uint64_t* cacheCost;22
23
-24
- analysis2PassFrameData* m_multipassAnalysis;25
- uint8_t* m_multipassDepth;26
- MV* m_multipassMv[2];27
- int* m_multipassMvpIdx[2];28
- int32_t* m_multipassRef[2];29
- uint8_t* m_multipassModes;30
-31
uint8_t m_evaluateInter;32
+ int32_t m_refineLevel;33
+34
uint8_t* m_additionalCtuInfo;35
int* m_prevCtuInfoChange;36
+37
+ struct TrainingData38
+ {39
+ uint32_t cuVariance;40
+ uint8_t predMode;41
+ uint8_t partSize;42
+ uint8_t mergeFlag;43
+ int split;44
+45
+ void init(const CUData& parentCTU, const CUGeom& cuGeom)46
+ {47
+ cuVariance = 0;48
+ predMode = parentCTU.m_predMode[cuGeom.absPartIdx];49
+ partSize = parentCTU.m_partSize[cuGeom.absPartIdx];50
+ mergeFlag = parentCTU.m_mergeFlag[cuGeom.absPartIdx];51
+ split = 0;52
+ }53
+ };54
+55
/* refine RD based on QP for rd-levels 5 and 6 */56
void qprdRefine(const CUData& parentCTU, const CUGeom& cuGeom, int32_t qp, int32_t lqp);57
58
59
void encodeResidue(const CUData& parentCTU, const CUGeom& cuGeom);60
61
int calculateQpforCuSize(const CUData& ctu, const CUGeom& cuGeom, int32_t complexCheck = 0, double baseQP = -1);62
+ uint32_t calculateCUVariance(const CUData& ctu, const CUGeom& cuGeom);63
+64
+ void classifyCU(const CUData& ctu, const CUGeom& cuGeom, const Mode& bestMode, TrainingData& trainData);65
+ void trainCU(const CUData& ctu, const CUGeom& cuGeom, const Mode& bestMode, TrainingData& trainData);66
67
void calculateNormFactor(CUData& ctu, int qp);68
void normFactor(const pixel* src, uint32_t blockSize, CUData& ctu, int qp, TextType ttype);69
x265_2.7.tar.gz/source/encoder/api.cpp -> x265_2.9.tar.gz/source/encoder/api.cpp
Changed
674
1
2
#include "nal.h"3
#include "bitcost.h"4
5
+#if ENABLE_LIBVMAF6
+#include "libvmaf.h"7
+#endif8
+9
/* multilib namespace reflectors */10
#if LINKED_8BIT11
namespace x265_8bit {12
13
pic_in->analysisData.wt = NULL;14
pic_in->analysisData.intraData = NULL;15
pic_in->analysisData.interData = NULL;16
- pic_in->analysis2Pass.analysisFramedata = NULL;17
+ pic_in->analysisData.distortionData = NULL;18
}19
20
- if (pp_nal && numEncoded > 0)21
+ if (pp_nal && numEncoded > 0 && encoder->m_outputCount >= encoder->m_latestParam->chunkStart)22
{23
*pp_nal = &encoder->m_nalList.m_nal[0];24
if (pi_nal) *pi_nal = encoder->m_nalList.m_numNal;25
26
else if (pi_nal)27
*pi_nal = 0;28
29
- if (numEncoded && encoder->m_param->csvLogLevel)30
+ if (numEncoded && encoder->m_param->csvLogLevel && encoder->m_outputCount >= encoder->m_latestParam->chunkStart)31
x265_csvlog_frame(encoder->m_param, pic_out);32
33
if (numEncoded < 0)34
35
encoder->fetchStats(outputStats, statsSizeBytes);36
}37
}38
+#if ENABLE_LIBVMAF39
+void x265_vmaf_encoder_log(x265_encoder* enc, int argc, char **argv, x265_param *param, x265_vmaf_data *vmafdata)40
+{41
+ if (enc)42
+ {43
+ Encoder *encoder = static_cast<Encoder*>(enc);44
+ x265_stats stats; 45
+ stats.aggregateVmafScore = x265_calculate_vmafscore(param, vmafdata);46
+ if(vmafdata->reference_file)47
+ fclose(vmafdata->reference_file);48
+ if(vmafdata->distorted_file)49
+ fclose(vmafdata->distorted_file);50
+ if(vmafdata)51
+ x265_free(vmafdata);52
+ encoder->fetchStats(&stats, sizeof(stats));53
+ int padx = encoder->m_sps.conformanceWindow.rightOffset;54
+ int pady = encoder->m_sps.conformanceWindow.bottomOffset;55
+ x265_csvlog_encode(encoder->m_param, &stats, padx, pady, argc, argv);56
+ }57
+}58
+#endif59
60
void x265_encoder_log(x265_encoder* enc, int argc, char **argv)61
{62
if (enc)63
{64
Encoder *encoder = static_cast<Encoder*>(enc);65
- x265_stats stats;66
+ x265_stats stats; 67
encoder->fetchStats(&stats, sizeof(stats));68
int padx = encoder->m_sps.conformanceWindow.rightOffset;69
int pady = encoder->m_sps.conformanceWindow.bottomOffset;70
71
return -1;72
}73
74
+void x265_alloc_analysis_data(x265_param *param, x265_analysis_data* analysis)75
+{76
+ x265_analysis_inter_data *interData = analysis->interData = NULL;77
+ x265_analysis_intra_data *intraData = analysis->intraData = NULL;78
+ x265_analysis_distortion_data *distortionData = analysis->distortionData = NULL;79
+ bool isVbv = param->rc.vbvMaxBitrate > 0 && param->rc.vbvBufferSize > 0;80
+ int numDir = 2; //irrespective of P or B slices set direction as 281
+ uint32_t numPlanes = param->internalCsp == X265_CSP_I400 ? 1 : 3;82
+83
+#if X265_DEPTH < 10 && (LINKED_10BIT || LINKED_12BIT)84
+ uint32_t numCUs_sse_t = param->internalBitDepth > 8 ? analysis->numCUsInFrame << 1 : analysis->numCUsInFrame;85
+#elif X265_DEPTH >= 10 && LINKED_8BIT86
+ uint32_t numCUs_sse_t = param->internalBitDepth > 8 ? analysis->numCUsInFrame : (analysis->numCUsInFrame + 1U) >> 1;87
+#else88
+ uint32_t numCUs_sse_t = analysis->numCUsInFrame;89
+#endif90
+91
+ //Allocate memory for distortionData pointer92
+ CHECKED_MALLOC_ZERO(distortionData, x265_analysis_distortion_data, 1);93
+ CHECKED_MALLOC_ZERO(distortionData->distortion, sse_t, analysis->numPartitions * numCUs_sse_t);94
+ if (param->rc.bStatRead)95
+ {96
+ CHECKED_MALLOC_ZERO(distortionData->ctuDistortion, sse_t, numCUs_sse_t);97
+ CHECKED_MALLOC_ZERO(distortionData->scaledDistortion, double, analysis->numCUsInFrame);98
+ CHECKED_MALLOC_ZERO(distortionData->offset, double, analysis->numCUsInFrame);99
+ CHECKED_MALLOC_ZERO(distortionData->threshold, double, analysis->numCUsInFrame);100
+ }101
+ analysis->distortionData = distortionData;102
+103
+ if (param->bDisableLookahead && isVbv)104
+ {105
+ CHECKED_MALLOC_ZERO(analysis->lookahead.intraSatdForVbv, uint32_t, analysis->numCuInHeight);106
+ CHECKED_MALLOC_ZERO(analysis->lookahead.satdForVbv, uint32_t, analysis->numCuInHeight);107
+ CHECKED_MALLOC_ZERO(analysis->lookahead.intraVbvCost, uint32_t, analysis->numCUsInFrame);108
+ CHECKED_MALLOC_ZERO(analysis->lookahead.vbvCost, uint32_t, analysis->numCUsInFrame);109
+ }110
+111
+ //Allocate memory for weightParam pointer112
+ if (!(param->bMVType == AVC_INFO))113
+ CHECKED_MALLOC_ZERO(analysis->wt, x265_weight_param, numPlanes * numDir);114
+115
+ if (param->analysisReuseLevel < 2)116
+ return;117
+118
+ //Allocate memory for intraData pointer119
+ CHECKED_MALLOC_ZERO(intraData, x265_analysis_intra_data, 1);120
+ CHECKED_MALLOC(intraData->depth, uint8_t, analysis->numPartitions * analysis->numCUsInFrame);121
+ CHECKED_MALLOC(intraData->modes, uint8_t, analysis->numPartitions * analysis->numCUsInFrame);122
+ CHECKED_MALLOC(intraData->partSizes, char, analysis->numPartitions * analysis->numCUsInFrame);123
+ CHECKED_MALLOC(intraData->chromaModes, uint8_t, analysis->numPartitions * analysis->numCUsInFrame);124
+ analysis->intraData = intraData;125
+126
+ //Allocate memory for interData pointer based on ReuseLevels127
+ CHECKED_MALLOC_ZERO(interData, x265_analysis_inter_data, 1);128
+ CHECKED_MALLOC(interData->depth, uint8_t, analysis->numPartitions * analysis->numCUsInFrame);129
+ CHECKED_MALLOC(interData->modes, uint8_t, analysis->numPartitions * analysis->numCUsInFrame);130
+131
+ CHECKED_MALLOC_ZERO(interData->mvpIdx[0], uint8_t, analysis->numPartitions * analysis->numCUsInFrame);132
+ CHECKED_MALLOC_ZERO(interData->mvpIdx[1], uint8_t, analysis->numPartitions * analysis->numCUsInFrame);133
+ CHECKED_MALLOC_ZERO(interData->mv[0], x265_analysis_MV, analysis->numPartitions * analysis->numCUsInFrame);134
+ CHECKED_MALLOC_ZERO(interData->mv[1], x265_analysis_MV, analysis->numPartitions * analysis->numCUsInFrame);135
+136
+ if (param->analysisReuseLevel > 4)137
+ {138
+ CHECKED_MALLOC(interData->partSize, uint8_t, analysis->numPartitions * analysis->numCUsInFrame);139
+ CHECKED_MALLOC_ZERO(interData->mergeFlag, uint8_t, analysis->numPartitions * analysis->numCUsInFrame);140
+ }141
+ if (param->analysisReuseLevel >= 7)142
+ {143
+ CHECKED_MALLOC(interData->interDir, uint8_t, analysis->numPartitions * analysis->numCUsInFrame);144
+ CHECKED_MALLOC(interData->sadCost, int64_t, analysis->numPartitions * analysis->numCUsInFrame);145
+ for (int dir = 0; dir < numDir; dir++)146
+ {147
+ CHECKED_MALLOC(interData->refIdx[dir], int8_t, analysis->numPartitions * analysis->numCUsInFrame);148
+ CHECKED_MALLOC_ZERO(analysis->modeFlag[dir], uint8_t, analysis->numPartitions * analysis->numCUsInFrame);149
+ }150
+ }151
+ else152
+ {153
+ if (param->analysisMultiPassRefine || param->analysisMultiPassDistortion){154
+ CHECKED_MALLOC_ZERO(interData->ref, int32_t, 2 * analysis->numPartitions * analysis->numCUsInFrame);155
+ }156
+ else157
+ CHECKED_MALLOC_ZERO(interData->ref, int32_t, analysis->numCUsInFrame * X265_MAX_PRED_MODE_PER_CTU * numDir);158
+ }159
+ analysis->interData = interData;160
+161
+ return;162
+163
+fail:164
+ x265_free_analysis_data(param, analysis);165
+}166
+167
+void x265_free_analysis_data(x265_param *param, x265_analysis_data* analysis)168
+{169
+ bool isVbv = param->rc.vbvMaxBitrate > 0 && param->rc.vbvBufferSize > 0;170
+171
+ //Free memory for Lookahead pointers172
+ if (param->bDisableLookahead && isVbv)173
+ {174
+ X265_FREE(analysis->lookahead.satdForVbv);175
+ X265_FREE(analysis->lookahead.intraSatdForVbv);176
+ X265_FREE(analysis->lookahead.vbvCost);177
+ X265_FREE(analysis->lookahead.intraVbvCost);178
+ }179
+180
+ //Free memory for distortionData pointers181
+ if (analysis->distortionData)182
+ {183
+ X265_FREE((analysis->distortionData)->distortion);184
+ if (param->rc.bStatRead)185
+ {186
+ X265_FREE((analysis->distortionData)->ctuDistortion);187
+ X265_FREE((analysis->distortionData)->scaledDistortion);188
+ X265_FREE((analysis->distortionData)->offset);189
+ X265_FREE((analysis->distortionData)->threshold);190
+ }191
+ X265_FREE(analysis->distortionData);192
+ }193
+194
+ /* Early exit freeing weights alone if level is 1 (when there is no analysis inter/intra) */195
+ if (analysis->wt && !(param->bMVType == AVC_INFO))196
+ X265_FREE(analysis->wt);197
+198
+ if (param->analysisReuseLevel < 2)199
+ return;200
+201
+ //Free memory for intraData pointers202
+ if (analysis->intraData)203
+ {204
+ X265_FREE((analysis->intraData)->depth);205
+ X265_FREE((analysis->intraData)->modes);206
+ X265_FREE((analysis->intraData)->partSizes);207
+ X265_FREE((analysis->intraData)->chromaModes);208
+ X265_FREE(analysis->intraData);209
+ analysis->intraData = NULL;210
+ }211
+212
+ //Free interData pointers213
+ if (analysis->interData)214
+ {215
+ X265_FREE((analysis->interData)->depth);216
+ X265_FREE((analysis->interData)->modes);217
+ X265_FREE((analysis->interData)->mvpIdx[0]);218
+ X265_FREE((analysis->interData)->mvpIdx[1]);219
+ X265_FREE((analysis->interData)->mv[0]);220
+ X265_FREE((analysis->interData)->mv[1]);221
+222
+ if (param->analysisReuseLevel > 4)223
+ {224
+ X265_FREE((analysis->interData)->mergeFlag);225
+ X265_FREE((analysis->interData)->partSize);226
+ }227
+ if (param->analysisReuseLevel >= 7)228
+ {229
+ int numDir = 2;230
+ X265_FREE((analysis->interData)->interDir);231
+ X265_FREE((analysis->interData)->sadCost);232
+ for (int dir = 0; dir < numDir; dir++)233
+ {234
+ X265_FREE((analysis->interData)->refIdx[dir]);235
+ if (analysis->modeFlag[dir] != NULL)236
+ {237
+ X265_FREE(analysis->modeFlag[dir]);238
+ analysis->modeFlag[dir] = NULL;239
+ }240
+ }241
+ }242
+ else243
+ X265_FREE((analysis->interData)->ref);244
+ X265_FREE(analysis->interData);245
+ analysis->interData = NULL;246
+ }247
+}248
+249
void x265_cleanup(void)250
{251
BitCost::destroy();252
253
&x265_csvlog_frame,254
&x265_csvlog_encode,255
&x265_dither_image,256
- &x265_set_analysis_data257
+ &x265_set_analysis_data,258
+#if ENABLE_LIBVMAF259
+ &x265_calculate_vmafscore,260
+ &x265_calculate_vmaf_framelevelscore,261
+ &x265_vmaf_encoder_log262
+#endif263
+264
};265
266
typedef const x265_api* (*api_get_func)(int bitDepth);267
268
if (param->rc.rateControlMode == X265_RC_CRF)269
fprintf(csvfp, "RateFactor, ");270
if (param->rc.vbvBufferSize)271
- fprintf(csvfp, "BufferFill, ");272
+ fprintf(csvfp, "BufferFill, BufferFillFinal, ");273
if (param->bEnablePsnr)274
fprintf(csvfp, "Y PSNR, U PSNR, V PSNR, YUV PSNR, ");275
if (param->bEnableSsim)276
277
/* detailed performance statistics */278
fprintf(csvfp, ", DecideWait (ms), Row0Wait (ms), Wall time (ms), Ref Wait Wall (ms), Total CTU time (ms),"279
"Stall Time (ms), Total frame time (ms), Avg WPP, Row Blocks");280
+#if ENABLE_LIBVMAF281
+ fprintf(csvfp, ", VMAF Frame Score");282
+#endif283
}284
fprintf(csvfp, "\n");285
}286
287
fputs(summaryCSVHeader, csvfp);288
if (param->csvLogLevel >= 2 || param->maxCLL || param->maxFALL)289
fputs("MaxCLL, MaxFALL,", csvfp);290
+#if ENABLE_LIBVMAF291
+ fputs(" Aggregate VMAF Score,", csvfp);292
+#endif293
fputs(" Version\n", csvfp);294
}295
}296
297
if (param->rc.rateControlMode == X265_RC_CRF)298
fprintf(param->csvfpt, "%.3lf,", frameStats->rateFactor);299
if (param->rc.vbvBufferSize)300
- fprintf(param->csvfpt, "%.3lf,", frameStats->bufferFill);301
+ fprintf(param->csvfpt, "%.3lf, %.3lf,", frameStats->bufferFill, frameStats->bufferFillFinal);302
if (param->bEnablePsnr)303
fprintf(param->csvfpt, "%.3lf, %.3lf, %.3lf, %.3lf,", frameStats->psnrY, frameStats->psnrU, frameStats->psnrV, frameStats->psnr);304
if (param->bEnableSsim)305
306
frameStats->totalFrameTime);307
308
fprintf(param->csvfpt, " %.3lf, %d", frameStats->avgWPP, frameStats->countRowBlocks);309
+#if ENABLE_LIBVMAF310
+ fprintf(param->csvfpt, ", %lf", frameStats->vmafFrameScore);311
+#endif312
}313
fprintf(param->csvfpt, "\n");314
fflush(stderr);315
316
fputs(summaryCSVHeader, p->csvfpt);317
if (p->csvLogLevel >= 2 || p->maxCLL || p->maxFALL)318
fputs("MaxCLL, MaxFALL,", p->csvfpt);319
+#if ENABLE_LIBVMAF320
+ fputs(" Aggregate VMAF score,", p->csvfpt);321
+#endif322
fputs(" Version\n",p->csvfpt);323
+324
}325
// CLI arguments or other326
if (argc)327
328
fputc('"', p->csvfpt);329
fputs(opts, p->csvfpt);330
fputc('"', p->csvfpt);331
+ X265_FREE(opts);332
}333
}334
335
336
char buffer[200];337
strftime(buffer, 128, "%c", timeinfo);338
fprintf(p->csvfpt, ", %s, ", buffer);339
-340
// elapsed time, fps, bitrate341
fprintf(p->csvfpt, "%.2f, %.2f, %.2f,",342
stats->elapsedEncodeTime, stats->encodedPictureCount / stats->elapsedEncodeTime, stats->bitrate);343
344
fprintf(p->csvfpt, " -, -, -, -, -, -, -,");345
if (p->csvLogLevel >= 2 || p->maxCLL || p->maxFALL)346
fprintf(p->csvfpt, " %-6u, %-6u,", stats->maxCLL, stats->maxFALL);347
+#if ENABLE_LIBVMAF348
+ fprintf(p->csvfpt, " %lf,", stats->aggregateVmafScore);349
+#endif350
fprintf(p->csvfpt, " %s\n", api->version_str);351
+352
}353
}354
355
356
}357
}358
359
+#if ENABLE_LIBVMAF360
+/* Read y values of single frame for 8-bit input */361
+int read_image_byte(FILE *file, float *buf, int width, int height, int stride)362
+{363
+ char *byte_ptr = (char *)buf;364
+ unsigned char *tmp_buf = 0;365
+ int i, j;366
+ int ret = 1;367
+368
+ if (width <= 0 || height <= 0)369
+ {370
+ goto fail_or_end;371
+ }372
+373
+ if (!(tmp_buf = (unsigned char*)malloc(width)))374
+ {375
+ goto fail_or_end;376
+ }377
+378
+ for (i = 0; i < height; ++i)379
+ {380
+ float *row_ptr = (float *)byte_ptr;381
+382
+ if (fread(tmp_buf, 1, width, file) != (size_t)width)383
+ {384
+ goto fail_or_end;385
+ }386
+387
+ for (j = 0; j < width; ++j)388
+ {389
+ row_ptr[j] = tmp_buf[j];390
+ }391
+392
+ byte_ptr += stride;393
+ }394
+395
+ ret = 0;396
+397
+fail_or_end:398
+ free(tmp_buf);399
+ return ret;400
+}401
+/* Read y values of single frame for 10-bit input */402
+int read_image_word(FILE *file, float *buf, int width, int height, int stride)403
+{404
+ char *byte_ptr = (char *)buf;405
+ unsigned short *tmp_buf = 0;406
+ int i, j;407
+ int ret = 1;408
+409
+ if (width <= 0 || height <= 0)410
+ {411
+ goto fail_or_end;412
+ }413
+414
+ if (!(tmp_buf = (unsigned short*)malloc(width * 2))) // '*2' to accommodate words415
+ {416
+ goto fail_or_end;417
+ }418
+419
+ for (i = 0; i < height; ++i)420
+ {421
+ float *row_ptr = (float *)byte_ptr;422
+423
+ if (fread(tmp_buf, 2, width, file) != (size_t)width) // '2' for word424
+ {425
+ goto fail_or_end;426
+ }427
+428
+ for (j = 0; j < width; ++j)429
+ {430
+ row_ptr[j] = tmp_buf[j] / 4.0; // '/4' to convert from 10 to 8-bit431
+ }432
+433
+ byte_ptr += stride;434
+ }435
+436
+ ret = 0;437
+438
+fail_or_end:439
+ free(tmp_buf);440
+ return ret;441
+}442
+443
+int read_frame(float *reference_data, float *distorted_data, float *temp_data, int stride_byte, void *s)444
+{445
+ x265_vmaf_data *user_data = (x265_vmaf_data *)s;446
+ int ret;447
+448
+ // read reference y449
+ if (user_data->internalBitDepth == 8)450
+ {451
+ ret = read_image_byte(user_data->reference_file, reference_data, user_data->width, user_data->height, stride_byte);452
+ }453
+ else if (user_data->internalBitDepth == 10)454
+ {455
+ ret = read_image_word(user_data->reference_file, reference_data, user_data->width, user_data->height, stride_byte);456
+ }457
+ else458
+ {459
+ x265_log(NULL, X265_LOG_ERROR, "Invalid bitdepth\n");460
+ return 1;461
+ }462
+ if (ret)463
+ {464
+ if (feof(user_data->reference_file))465
+ {466
+ ret = 2; // OK if end of file467
+ }468
+ return ret;469
+ }470
+471
+ // read distorted y472
+ if (user_data->internalBitDepth == 8)473
+ {474
+ ret = read_image_byte(user_data->distorted_file, distorted_data, user_data->width, user_data->height, stride_byte);475
+ }476
+ else if (user_data->internalBitDepth == 10)477
+ {478
+ ret = read_image_word(user_data->distorted_file, distorted_data, user_data->width, user_data->height, stride_byte);479
+ }480
+ else481
+ {482
+ x265_log(NULL, X265_LOG_ERROR, "Invalid bitdepth\n");483
+ return 1;484
+ }485
+ if (ret)486
+ {487
+ if (feof(user_data->distorted_file))488
+ {489
+ ret = 2; // OK if end of file490
+ }491
+ return ret;492
+ }493
+494
+ // reference skip u and v495
+ if (user_data->internalBitDepth == 8)496
+ {497
+ if (fread(temp_data, 1, user_data->offset, user_data->reference_file) != (size_t)user_data->offset)498
+ {499
+ x265_log(NULL, X265_LOG_ERROR, "reference fread to skip u and v failed.\n");500
+ goto fail_or_end;501
+ }502
+ }503
+ else if (user_data->internalBitDepth == 10)504
+ {505
+ if (fread(temp_data, 2, user_data->offset, user_data->reference_file) != (size_t)user_data->offset)506
+ {507
+ x265_log(NULL, X265_LOG_ERROR, "reference fread to skip u and v failed.\n");508
+ goto fail_or_end;509
+ }510
+ }511
+ else512
+ {513
+ x265_log(NULL, X265_LOG_ERROR, "Invalid format\n");514
+ goto fail_or_end;515
+ }516
+517
+ // distorted skip u and v518
+ if (user_data->internalBitDepth == 8)519
+ {520
+ if (fread(temp_data, 1, user_data->offset, user_data->distorted_file) != (size_t)user_data->offset)521
+ {522
+ x265_log(NULL, X265_LOG_ERROR, "distorted fread to skip u and v failed.\n");523
+ goto fail_or_end;524
+ }525
+ }526
+ else if (user_data->internalBitDepth == 10)527
+ {528
+ if (fread(temp_data, 2, user_data->offset, user_data->distorted_file) != (size_t)user_data->offset)529
+ {530
+ x265_log(NULL, X265_LOG_ERROR, "distorted fread to skip u and v failed.\n");531
+ goto fail_or_end;532
+ }533
+ }534
+ else535
+ {536
+ x265_log(NULL, X265_LOG_ERROR, "Invalid format\n");537
+ goto fail_or_end;538
+ }539
+540
+541
+fail_or_end:542
+ return ret;543
+}544
+545
+double x265_calculate_vmafscore(x265_param *param, x265_vmaf_data *data)546
+{547
+ double score;548
+ 549
+ data->width = param->sourceWidth;550
+ data->height = param->sourceHeight;551
+ data->internalBitDepth = param->internalBitDepth;552
+ 553
+ if (param->internalCsp == X265_CSP_I420)554
+ {555
+ if ((param->sourceWidth * param->sourceHeight) % 2 != 0)556
+ x265_log(NULL, X265_LOG_ERROR, "Invalid file size\n");557
+ data->offset = param->sourceWidth * param->sourceHeight / 2;558
+ }559
+ else if (param->internalCsp == X265_CSP_I422)560
+ data->offset = param->sourceWidth * param->sourceHeight;561
+ else if (param->internalCsp == X265_CSP_I444)562
+ data->offset = param->sourceWidth * param->sourceHeight * 2;563
+ else564
+ x265_log(NULL, X265_LOG_ERROR, "Invalid format\n");565
+ 566
+ compute_vmaf(&score, vcd->format, data->width, data->height, read_frame, data, vcd->model_path, vcd->log_path, vcd->log_fmt, vcd->disable_clip, vcd->disable_avx, vcd->enable_transform, vcd->phone_model, vcd->psnr, vcd->ssim, vcd->ms_ssim, vcd->pool); 567
+568
+ return score;569
+}570
+571
+int read_frame_10bit(float *reference_data, float *distorted_data, float *temp_data, int stride, void *s)572
+{573
+ x265_vmaf_framedata *user_data = (x265_vmaf_framedata *)s;574
+575
+ PicYuv *reference_frame = (PicYuv *)user_data->reference_frame;576
+ PicYuv *distorted_frame = (PicYuv *)user_data->distorted_frame;577
+578
+ if(!user_data->frame_set) {579
+ 580
+ int reference_stride = reference_frame->m_stride;581
+ int distorted_stride = distorted_frame->m_stride;582
+583
+ const uint16_t *reference_ptr = (const uint16_t *)reference_frame->m_picOrg[0]; 584
+ const uint16_t *distorted_ptr = (const uint16_t *)distorted_frame->m_picOrg[0];585
+586
+ temp_data = reference_data;587
+588
+ int height = user_data->height;589
+ int width = user_data->width; 590
+591
+ int i,j;592
+ for (i = 0; i < height; i++) {593
+ for ( j = 0; j < width; j++) {594
+ temp_data[j] = ((float)reference_ptr[j] / 4.0);595
+ }596
+ reference_ptr += reference_stride;597
+ temp_data += stride / sizeof(*temp_data);598
+ }599
+ 600
+ temp_data = distorted_data;601
+ for (i = 0; i < height; i++) {602
+ for (j = 0; j < width; j++) {603
+ temp_data[j] = ((float)distorted_ptr[j] / 4.0);604
+ }605
+ distorted_ptr += distorted_stride;606
+ temp_data += stride / sizeof(*temp_data);607
+ }608
+609
+ user_data->frame_set = 1;610
+ return 0;611
+ } 612
+ return 2; 613
+}614
+615
+int read_frame_8bit(float *reference_data, float *distorted_data, float *temp_data, int stride, void *s)616
+{617
+ x265_vmaf_framedata *user_data = (x265_vmaf_framedata *)s;618
+619
+ PicYuv *reference_frame = (PicYuv *)user_data->reference_frame;620
+ PicYuv *distorted_frame = (PicYuv *)user_data->distorted_frame;621
+622
+ if(!user_data->frame_set) {623
+624
+ int reference_stride = reference_frame->m_stride;625
+ int distorted_stride = distorted_frame->m_stride;626
+627
+ const uint8_t *reference_ptr = (const uint8_t *)reference_frame->m_picOrg[0]; 628
+ const uint8_t *distorted_ptr = (const uint8_t *)distorted_frame->m_picOrg[0];629
+630
+ temp_data = reference_data;631
+632
+ int height = user_data->height;633
+ int width = user_data->width; 634
+635
+ int i,j;636
+ for (i = 0; i < height; i++) {637
+ for ( j = 0; j < width; j++) {638
+ temp_data[j] = (float)reference_ptr[j];639
+ }640
+ reference_ptr += reference_stride;641
+ temp_data += stride / sizeof(*temp_data);642
+ }643
+ 644
+ temp_data = distorted_data;645
+ for (i = 0; i < height; i++) {646
+ for (j = 0; j < width; j++) {647
+ temp_data[j] = (float)distorted_ptr[j];648
+ }649
+ distorted_ptr += distorted_stride;650
+ temp_data += stride / sizeof(*temp_data);651
+ }652
+653
+ user_data->frame_set = 1;654
+ return 0;655
+ } 656
+ return 2; 657
+}658
+659
+double x265_calculate_vmaf_framelevelscore(x265_vmaf_framedata *vmafframedata)660
+{661
+ double score; 662
+ int (*read_frame)(float *reference_data, float *distorted_data, float *temp_data,663
+ int stride, void *s);664
+ if (vmafframedata->internalBitDepth == 8)665
+ read_frame = read_frame_8bit;666
+ else667
+ read_frame = read_frame_10bit;668
+ compute_vmaf(&score, vcd->format, vmafframedata->width, vmafframedata->height, read_frame, vmafframedata, vcd->model_path, vcd->log_path, vcd->log_fmt, vcd->disable_clip, vcd->disable_avx, vcd->enable_transform, vcd->phone_model, vcd->psnr, vcd->ssim, vcd->ms_ssim, vcd->pool);669
+ 670
+ return score;671
+}672
+#endif673
} /* end namespace or extern "C" */674
x265_2.7.tar.gz/source/encoder/dpb.cpp -> x265_2.9.tar.gz/source/encoder/dpb.cpp
Changed
35
1
2
int pocCurr = slice->m_poc;3
int type = newFrame->m_lowres.sliceType;4
bool bIsKeyFrame = newFrame->m_lowres.bKeyframe;5
-6
slice->m_nalUnitType = getNalUnitType(pocCurr, bIsKeyFrame);7
- if (slice->m_nalUnitType == NAL_UNIT_CODED_SLICE_IDR_W_RADL)8
+ if (slice->m_nalUnitType == NAL_UNIT_CODED_SLICE_IDR_W_RADL || slice->m_nalUnitType == NAL_UNIT_CODED_SLICE_IDR_N_LP)9
m_lastIDR = pocCurr;10
slice->m_lastIDR = m_lastIDR;11
slice->m_sliceType = IS_X265_TYPE_B(type) ? B_SLICE : (type == X265_TYPE_P) ? P_SLICE : I_SLICE;12
13
/* Marking reference pictures when an IDR/CRA is encountered. */14
void DPB::decodingRefreshMarking(int pocCurr, NalUnitType nalUnitType)15
{16
- if (nalUnitType == NAL_UNIT_CODED_SLICE_IDR_W_RADL)17
+ if (nalUnitType == NAL_UNIT_CODED_SLICE_IDR_W_RADL || nalUnitType == NAL_UNIT_CODED_SLICE_IDR_N_LP)18
{19
/* If the nal_unit_type is IDR, all pictures in the reference picture20
* list are marked as "unused for reference" */21
22
NalUnitType DPB::getNalUnitType(int curPOC, bool bIsKeyFrame)23
{24
if (!curPOC)25
- return NAL_UNIT_CODED_SLICE_IDR_W_RADL;26
-27
+ return NAL_UNIT_CODED_SLICE_IDR_N_LP;28
if (bIsKeyFrame)29
- return m_bOpenGOP ? NAL_UNIT_CODED_SLICE_CRA : NAL_UNIT_CODED_SLICE_IDR_W_RADL;30
-31
+ return m_bOpenGOP ? NAL_UNIT_CODED_SLICE_CRA : m_bhasLeadingPicture ? NAL_UNIT_CODED_SLICE_IDR_W_RADL : NAL_UNIT_CODED_SLICE_IDR_N_LP;32
if (m_pocCRA && curPOC < m_pocCRA)33
// All leading pictures are being marked as TFD pictures here since34
// current encoder uses all reference pictures while encoding leading35
x265_2.7.tar.gz/source/encoder/dpb.h -> x265_2.9.tar.gz/source/encoder/dpb.h
Changed
17
1
2
int m_lastIDR;3
int m_pocCRA;4
int m_bOpenGOP;5
+ int m_bhasLeadingPicture;6
bool m_bRefreshPending;7
bool m_bTemporalSublayer;8
PicList m_picList;9
10
{11
m_lastIDR = 0;12
m_pocCRA = 0;13
+ m_bhasLeadingPicture = param->radl;14
m_bRefreshPending = false;15
m_frameDataFreeList = NULL;16
m_bOpenGOP = param->bOpenGOP;17
x265_2.7.tar.gz/source/encoder/encoder.cpp -> x265_2.9.tar.gz/source/encoder/encoder.cpp
Changed
2234
1
2
m_threadPool = NULL;3
m_analysisFileIn = NULL;4
m_analysisFileOut = NULL;5
+ m_naluFile = NULL;6
m_offsetEmergency = NULL;7
m_iFrameNum = 0;8
m_iPPSQpMinus26 = 0;9
10
#endif11
12
m_prevTonemapPayload.payload = NULL;13
+ m_startPoint = 0;14
+ m_saveCTUSize = 0;15
}16
inline char *strcatFilename(const char *input, const char *suffix)17
{18
19
20
if (m_param->bEmitHRDSEI)21
m_rateControl->initHRD(m_sps);22
+23
if (!m_rateControl->init(m_sps))24
m_aborted = true;25
if (!m_lookahead->create())26
m_aborted = true;27
+28
initRefIdx();29
if (m_param->analysisSave && m_param->bUseAnalysisFile)30
{31
32
33
m_emitCLLSEI = p->maxCLL || p->maxFALL;34
35
+ if (m_param->naluFile)36
+ {37
+ m_naluFile = x265_fopen(m_param->naluFile, "r");38
+ if (!m_naluFile)39
+ {40
+ x265_log_file(NULL, X265_LOG_ERROR, "%s file not found or Failed to open\n", m_param->naluFile);41
+ m_aborted = true;42
+ }43
+ else44
+ m_enableNal = 1;45
+ }46
+ else47
+ m_enableNal = 0;48
+49
#if ENABLE_HDR10_PLUS50
if (m_bToneMap)51
m_numCimInfo = m_hdr10plus_api->hdr10plus_json_to_movie_cim(m_param->toneMapFile, m_cim);52
#endif53
+ if (m_param->bDynamicRefine)54
+ {55
+ /* Allocate memory for 1 GOP and reuse it for the subsequent GOPs */56
+ int size = (m_param->keyframeMax + m_param->lookaheadDepth) * m_param->maxCUDepth * X265_REFINE_INTER_LEVELS;57
+ CHECKED_MALLOC_ZERO(m_variance, uint64_t, size);58
+ CHECKED_MALLOC_ZERO(m_rdCost, uint64_t, size);59
+ CHECKED_MALLOC_ZERO(m_trainingCount, uint32_t, size);60
+ return;61
+ fail:62
+ m_aborted = true;63
+ }64
}65
66
void Encoder::stopJobs()67
68
curFrame->m_analysisData.numPartitions = m_param->num4x4Partitions;69
int num16x16inCUWidth = m_param->maxCUSize >> 4;70
uint32_t ctuAddr, offset, cuPos;71
- analysis_intra_data * intraData = (analysis_intra_data *)curFrame->m_analysisData.intraData;72
- analysis_intra_data * srcIntraData = (analysis_intra_data *)analysis_data->intraData;73
+ x265_analysis_intra_data * intraData = curFrame->m_analysisData.intraData;74
+ x265_analysis_intra_data * srcIntraData = analysis_data->intraData;75
for (int i = 0; i < mbImageHeight; i++)76
{77
for (int j = 0; j < mbImageWidth; j++)78
79
curFrame->m_analysisData.numPartitions = m_param->num4x4Partitions;80
int num16x16inCUWidth = m_param->maxCUSize >> 4;81
uint32_t ctuAddr, offset, cuPos;82
- analysis_inter_data * interData = (analysis_inter_data *)curFrame->m_analysisData.interData;83
- analysis_inter_data * srcInterData = (analysis_inter_data*)analysis_data->interData;84
+ x265_analysis_inter_data * interData = curFrame->m_analysisData.interData;85
+ x265_analysis_inter_data * srcInterData = analysis_data->interData;86
for (int i = 0; i < mbImageHeight; i++)87
{88
for (int j = 0; j < mbImageWidth; j++)89
90
curFrame->m_analysisData = (*analysis_data);91
curFrame->m_analysisData.numCUsInFrame = widthInCU * heightInCU;92
curFrame->m_analysisData.numPartitions = m_param->num4x4Partitions;93
- allocAnalysis(&curFrame->m_analysisData);94
+ x265_alloc_analysis_data(m_param, &curFrame->m_analysisData);95
if (m_param->maxCUSize == 16)96
{97
if (analysis_data->sliceType == X265_TYPE_IDR || analysis_data->sliceType == X265_TYPE_I)98
99
100
curFrame->m_analysisData.numPartitions = m_param->num4x4Partitions;101
size_t count = 0;102
- analysis_intra_data * currIntraData = (analysis_intra_data *)curFrame->m_analysisData.intraData;103
- analysis_intra_data * intraData = (analysis_intra_data *)analysis_data->intraData;104
+ x265_analysis_intra_data * currIntraData = curFrame->m_analysisData.intraData;105
+ x265_analysis_intra_data * intraData = analysis_data->intraData;106
for (uint32_t d = 0; d < cuBytes; d++)107
{108
int bytes = curFrame->m_analysisData.numPartitions >> ((intraData)->depth[d] * 2);109
110
111
curFrame->m_analysisData.numPartitions = m_param->num4x4Partitions;112
size_t count = 0;113
- analysis_inter_data * currInterData = (analysis_inter_data *)curFrame->m_analysisData.interData;114
- analysis_inter_data * interData = (analysis_inter_data *)analysis_data->interData;115
+ x265_analysis_inter_data * currInterData = curFrame->m_analysisData.interData;116
+ x265_analysis_inter_data * interData = analysis_data->interData;117
for (uint32_t d = 0; d < cuBytes; d++)118
{119
int bytes = curFrame->m_analysisData.numPartitions >> ((interData)->depth[d] * 2);120
memset(&(currInterData)->depth[count], (interData)->depth[d], bytes);121
memset(&(currInterData)->modes[count], (interData)->modes[d], bytes);122
- memcpy(&(currInterData)->sadCost[count], &((analysis_inter_data*)analysis_data->interData)->sadCost[d], bytes);123
+ memcpy(&(currInterData)->sadCost[count], &(analysis_data->interData)->sadCost[d], bytes);124
if (m_param->analysisReuseLevel > 4)125
{126
memset(&(currInterData)->partSize[count], (interData)->partSize[d], bytes);127
128
if (m_bToneMap)129
m_hdr10plus_api->hdr10plus_clear_movie(m_cim, m_numCimInfo);130
#endif131
- 132
+133
+ if (m_param->bDynamicRefine)134
+ {135
+ X265_FREE(m_variance);136
+ X265_FREE(m_rdCost);137
+ X265_FREE(m_trainingCount);138
+ }139
if (m_exportedPic)140
{141
ATOMIC_DEC(&m_exportedPic->m_countRefEncoders);142
143
}144
X265_FREE(temp);145
}146
+ if (m_naluFile)147
+ fclose(m_naluFile);148
if (m_param)149
{150
if (m_param->csvfpt)151
152
}153
}154
155
+void Encoder::copyUserSEIMessages(Frame *frame, const x265_picture* pic_in)156
+{157
+ x265_sei_payload toneMap;158
+ toneMap.payload = NULL;159
+ int toneMapPayload = 0;160
+161
+#if ENABLE_HDR10_PLUS162
+ if (m_bToneMap)163
+ {164
+ int currentPOC = m_pocLast;165
+ if (currentPOC < m_numCimInfo)166
+ {167
+ int32_t i = 0;168
+ toneMap.payloadSize = 0;169
+ while (m_cim[currentPOC][i] == 0xFF)170
+ toneMap.payloadSize += m_cim[currentPOC][i++];171
+ toneMap.payloadSize += m_cim[currentPOC][i];172
+173
+ toneMap.payload = (uint8_t*)x265_malloc(sizeof(uint8_t) * toneMap.payloadSize);174
+ toneMap.payloadType = USER_DATA_REGISTERED_ITU_T_T35;175
+ memcpy(toneMap.payload, &m_cim[currentPOC][i + 1], toneMap.payloadSize);176
+ toneMapPayload = 1;177
+ }178
+ }179
+#endif180
+ /* seiMsg will contain SEI messages specified in a fixed file format in POC order.181
+ * Format of the file : <POC><space><PREFIX><space><NAL UNIT TYPE>/<SEI TYPE><space><SEI Payload> */182
+ x265_sei_payload seiMsg;183
+ seiMsg.payload = NULL;184
+ int userPayload = 0;185
+ if (m_enableNal)186
+ {187
+ readUserSeiFile(seiMsg, m_pocLast);188
+ if (seiMsg.payload)189
+ userPayload = 1;;190
+ }191
+192
+ int numPayloads = pic_in->userSEI.numPayloads + toneMapPayload + userPayload;193
+ frame->m_userSEI.numPayloads = numPayloads;194
+195
+ if (frame->m_userSEI.numPayloads)196
+ {197
+ if (!frame->m_userSEI.payloads)198
+ {199
+ frame->m_userSEI.payloads = new x265_sei_payload[numPayloads];200
+ for (int i = 0; i < numPayloads; i++)201
+ frame->m_userSEI.payloads[i].payload = NULL;202
+ }203
+ for (int i = 0; i < numPayloads; i++)204
+ {205
+ x265_sei_payload input;206
+ if ((i == (numPayloads - 1)) && toneMapPayload)207
+ input = toneMap;208
+ else if (m_enableNal)209
+ input = seiMsg;210
+ else211
+ input = pic_in->userSEI.payloads[i];212
+213
+ if (!frame->m_userSEI.payloads[i].payload)214
+ frame->m_userSEI.payloads[i].payload = new uint8_t[input.payloadSize];215
+ memcpy(frame->m_userSEI.payloads[i].payload, input.payload, input.payloadSize);216
+ frame->m_userSEI.payloads[i].payloadSize = input.payloadSize;217
+ frame->m_userSEI.payloads[i].payloadType = input.payloadType;218
+ }219
+ if (toneMap.payload)220
+ x265_free(toneMap.payload);221
+ if (seiMsg.payload)222
+ x265_free(seiMsg.payload);223
+ }224
+}225
+226
/**227
* Feed one new input frame into the encoder, get one frame out. If pic_in is228
* NULL, a flush condition is implied and pic_in must be NULL for all subsequent229
230
if (m_exportedPic)231
{232
if (!m_param->bUseAnalysisFile && m_param->analysisSave)233
- freeAnalysis(&m_exportedPic->m_analysisData);234
+ x265_free_analysis_data(m_param, &m_exportedPic->m_analysisData);235
ATOMIC_DEC(&m_exportedPic->m_countRefEncoders);236
m_exportedPic = NULL;237
m_dpb->recycleUnreferenced();238
}239
- if (pic_in)240
+ if (pic_in && (!m_param->chunkEnd || (m_encodedFrameNum < m_param->chunkEnd)))241
{242
if (m_latestParam->forceFlush == 1)243
{244
245
m_latestParam->forceFlush = 0;246
}247
248
- x265_sei_payload toneMap;249
- toneMap.payload = NULL;250
-#if ENABLE_HDR10_PLUS251
- if (m_bToneMap)252
- {253
- int currentPOC = m_pocLast + 1;254
- if (currentPOC < m_numCimInfo)255
- {256
- int32_t i = 0;257
- toneMap.payloadSize = 0;258
- while (m_cim[currentPOC][i] == 0xFF)259
- toneMap.payloadSize += m_cim[currentPOC][i++];260
- toneMap.payloadSize += m_cim[currentPOC][i];261
-262
- toneMap.payload = (uint8_t*)x265_malloc(sizeof(uint8_t) * toneMap.payloadSize);263
- toneMap.payloadType = USER_DATA_REGISTERED_ITU_T_T35;264
- memcpy(toneMap.payload, &m_cim[currentPOC][i+1], toneMap.payloadSize);265
- }266
- }267
-#endif268
-269
if (pic_in->bitDepth < 8 || pic_in->bitDepth > 16)270
{271
x265_log(m_param, X265_LOG_ERROR, "Input bit depth (%d) must be between 8 and 16\n",272
273
inFrame->m_forceqp = pic_in->forceqp;274
inFrame->m_param = (m_reconfigure || m_reconfigureRc) ? m_latestParam : m_param;275
276
- int toneMapEnable = 0;277
- if (m_bToneMap && toneMap.payload)278
- toneMapEnable = 1;279
- int numPayloads = pic_in->userSEI.numPayloads + toneMapEnable;280
- inFrame->m_userSEI.numPayloads = numPayloads;281
-282
- if (inFrame->m_userSEI.numPayloads)283
- {284
- if (!inFrame->m_userSEI.payloads)285
- {286
- inFrame->m_userSEI.payloads = new x265_sei_payload[numPayloads];287
- for (int i = 0; i < numPayloads; i++)288
- inFrame->m_userSEI.payloads[i].payload = NULL;289
- }290
- for (int i = 0; i < numPayloads; i++)291
- {292
- x265_sei_payload input;293
- if ((i == (numPayloads - 1)) && toneMapEnable)294
- input = toneMap;295
- else296
- input = pic_in->userSEI.payloads[i];297
- int size = inFrame->m_userSEI.payloads[i].payloadSize = input.payloadSize;298
- inFrame->m_userSEI.payloads[i].payloadType = input.payloadType;299
- if (!inFrame->m_userSEI.payloads[i].payload)300
- inFrame->m_userSEI.payloads[i].payload = new uint8_t[size];301
- memcpy(inFrame->m_userSEI.payloads[i].payload, input.payload, size);302
- }303
- if (toneMap.payload)304
- x265_free(toneMap.payload);305
- }306
+ copyUserSEIMessages(inFrame, pic_in);307
308
if (pic_in->quantOffsets != NULL)309
{310
311
/* Load analysis data before lookahead->addPicture, since sliceType has been decided */312
if (m_param->analysisLoad)313
{314
- /* readAnalysisFile reads analysis data for the frame and allocates memory based on slicetype */315
- readAnalysisFile(&inFrame->m_analysisData, inFrame->m_poc, pic_in);316
+ /* reads analysis data for the frame and allocates memory based on slicetype */317
+ static int paramBytes = 0;318
+ if (!inFrame->m_poc)319
+ {320
+ x265_analysis_data analysisData = pic_in->analysisData;321
+ paramBytes = validateAnalysisData(&analysisData, 0);322
+ if (paramBytes == -1)323
+ {324
+ m_aborted = true;325
+ return -1;326
+ }327
+ }328
+ if (m_saveCTUSize)329
+ {330
+ cuLocation cuLocInFrame;331
+ cuLocInFrame.init(m_param);332
+ /* Set skipWidth/skipHeight flags when the out of bound pixels in lowRes is greater than half of maxCUSize */333
+ int extendedWidth = ((m_param->sourceWidth / 2 + m_param->maxCUSize - 1) >> m_param->maxLog2CUSize) * m_param->maxCUSize;334
+ int extendedHeight = ((m_param->sourceHeight / 2 + m_param->maxCUSize - 1) >> m_param->maxLog2CUSize) * m_param->maxCUSize;335
+ uint32_t outOfBoundaryLowres = extendedWidth - m_param->sourceWidth / 2;336
+ if (outOfBoundaryLowres * 2 >= m_param->maxCUSize)337
+ cuLocInFrame.skipWidth = true;338
+ uint32_t outOfBoundaryLowresH = extendedHeight - m_param->sourceHeight / 2;339
+ if (outOfBoundaryLowresH * 2 >= m_param->maxCUSize)340
+ cuLocInFrame.skipHeight = true;341
+ readAnalysisFile(&inFrame->m_analysisData, inFrame->m_poc, pic_in, paramBytes, cuLocInFrame);342
+ }343
+ else344
+ readAnalysisFile(&inFrame->m_analysisData, inFrame->m_poc, pic_in, paramBytes);345
inFrame->m_poc = inFrame->m_analysisData.poc;346
sliceType = inFrame->m_analysisData.sliceType;347
inFrame->m_lowres.bScenecut = !!inFrame->m_analysisData.bScenecut;348
349
350
/* Free up pic_in->analysisData since it has already been used */351
if ((m_param->analysisLoad && !m_param->analysisSave) || (m_param->bMVType && slice->m_sliceType != I_SLICE))352
- freeAnalysis(&outFrame->m_analysisData);353
+ x265_free_analysis_data(m_param, &outFrame->m_analysisData);354
355
if (pic_out)356
{357
358
359
pic_out->pts = outFrame->m_pts;360
pic_out->dts = outFrame->m_dts;361
+ pic_out->reorderedPts = outFrame->m_reorderedPts;362
pic_out->sliceType = outFrame->m_lowres.sliceType;363
pic_out->planes[0] = recpic->m_picOrg[0];364
pic_out->stride[0] = (int)(recpic->m_stride * sizeof(pixel));365
366
pic_out->analysisData.intraData = outFrame->m_analysisData.intraData;367
pic_out->analysisData.modeFlag[0] = outFrame->m_analysisData.modeFlag[0];368
pic_out->analysisData.modeFlag[1] = outFrame->m_analysisData.modeFlag[1];369
+ pic_out->analysisData.distortionData = outFrame->m_analysisData.distortionData;370
if (m_param->bDisableLookahead)371
{372
int factor = 1;373
374
factor = m_param->scaleFactor * 2;375
pic_out->analysisData.numCuInHeight = outFrame->m_analysisData.numCuInHeight;376
pic_out->analysisData.lookahead.dts = outFrame->m_dts;377
+ pic_out->analysisData.lookahead.reorderedPts = outFrame->m_reorderedPts;378
pic_out->analysisData.satdCost *= factor;379
pic_out->analysisData.lookahead.keyframe = outFrame->m_lowres.bKeyframe;380
pic_out->analysisData.lookahead.lastMiniGopBFrame = outFrame->m_lowres.bLastMiniGopBFrame;381
382
int vbvCount = m_param->lookaheadDepth + m_param->bframes + 2;383
for (int index = 0; index < vbvCount; index++)384
{385
- pic_out->analysisData.lookahead.plannedSatd[index] = outFrame->m_lowres.plannedSatd[index] * factor;386
+ pic_out->analysisData.lookahead.plannedSatd[index] = outFrame->m_lowres.plannedSatd[index];387
pic_out->analysisData.lookahead.plannedType[index] = outFrame->m_lowres.plannedType[index];388
}389
for (uint32_t index = 0; index < pic_out->analysisData.numCuInHeight; index++)390
{391
- outFrame->m_analysisData.lookahead.intraSatdForVbv[index] = outFrame->m_encData->m_rowStat[index].intraSatdForVbv * factor;392
- outFrame->m_analysisData.lookahead.satdForVbv[index] = outFrame->m_encData->m_rowStat[index].satdForVbv * factor;393
+ outFrame->m_analysisData.lookahead.intraSatdForVbv[index] = outFrame->m_encData->m_rowStat[index].intraSatdForVbv;394
+ outFrame->m_analysisData.lookahead.satdForVbv[index] = outFrame->m_encData->m_rowStat[index].satdForVbv;395
}396
pic_out->analysisData.lookahead.intraSatdForVbv = outFrame->m_analysisData.lookahead.intraSatdForVbv;397
pic_out->analysisData.lookahead.satdForVbv = outFrame->m_analysisData.lookahead.satdForVbv;398
for (uint32_t index = 0; index < pic_out->analysisData.numCUsInFrame; index++)399
{400
- outFrame->m_analysisData.lookahead.intraVbvCost[index] = outFrame->m_encData->m_cuStat[index].intraVbvCost * factor;401
- outFrame->m_analysisData.lookahead.vbvCost[index] = outFrame->m_encData->m_cuStat[index].vbvCost * factor;402
+ outFrame->m_analysisData.lookahead.intraVbvCost[index] = outFrame->m_encData->m_cuStat[index].intraVbvCost;403
+ outFrame->m_analysisData.lookahead.vbvCost[index] = outFrame->m_encData->m_cuStat[index].vbvCost;404
}405
pic_out->analysisData.lookahead.intraVbvCost = outFrame->m_analysisData.lookahead.intraVbvCost;406
pic_out->analysisData.lookahead.vbvCost = outFrame->m_analysisData.lookahead.vbvCost;407
}408
}409
writeAnalysisFile(&pic_out->analysisData, *outFrame->m_encData);410
+ pic_out->analysisData.saveParam = pic_out->analysisData.saveParam;411
if (m_param->bUseAnalysisFile)412
- freeAnalysis(&pic_out->analysisData);413
+ x265_free_analysis_data(m_param, &pic_out->analysisData);414
}415
}416
if (m_param->rc.bStatWrite && (m_param->analysisMultiPassRefine || m_param->analysisMultiPassDistortion))417
{418
if (pic_out)419
{420
- pic_out->analysis2Pass.poc = pic_out->poc;421
- pic_out->analysis2Pass.analysisFramedata = outFrame->m_analysis2Pass.analysisFramedata;422
+ pic_out->analysisData.poc = pic_out->poc;423
+ pic_out->analysisData.interData = outFrame->m_analysisData.interData;424
+ pic_out->analysisData.intraData = outFrame->m_analysisData.intraData;425
+ pic_out->analysisData.distortionData = outFrame->m_analysisData.distortionData;426
}427
- writeAnalysis2PassFile(&outFrame->m_analysis2Pass, *outFrame->m_encData, outFrame->m_lowres.sliceType);428
+ writeAnalysisFileRefine(&outFrame->m_analysisData, *outFrame->m_encData);429
}430
if (m_param->analysisMultiPassRefine || m_param->analysisMultiPassDistortion)431
- freeAnalysis2Pass(&outFrame->m_analysis2Pass, outFrame->m_lowres.sliceType);432
+ x265_free_analysis_data(m_param, &outFrame->m_analysisData);433
if (m_param->internalCsp == X265_CSP_I400)434
{435
if (slice->m_sliceType == P_SLICE)436
{437
- if (slice->m_weightPredTable[0][0][0].bPresentFlag)438
+ if (slice->m_weightPredTable[0][0][0].wtPresent)439
m_numLumaWPFrames++;440
}441
else if (slice->m_sliceType == B_SLICE)442
443
bool bLuma = false;444
for (int l = 0; l < 2; l++)445
{446
- if (slice->m_weightPredTable[l][0][0].bPresentFlag)447
+ if (slice->m_weightPredTable[l][0][0].wtPresent)448
bLuma = true;449
}450
if (bLuma)451
452
{453
if (slice->m_sliceType == P_SLICE)454
{455
- if (slice->m_weightPredTable[0][0][0].bPresentFlag)456
+ if (slice->m_weightPredTable[0][0][0].wtPresent)457
m_numLumaWPFrames++;458
- if (slice->m_weightPredTable[0][0][1].bPresentFlag ||459
- slice->m_weightPredTable[0][0][2].bPresentFlag)460
+ if (slice->m_weightPredTable[0][0][1].wtPresent ||461
+ slice->m_weightPredTable[0][0][2].wtPresent)462
m_numChromaWPFrames++;463
}464
else if (slice->m_sliceType == B_SLICE)465
466
bool bLuma = false, bChroma = false;467
for (int l = 0; l < 2; l++)468
{469
- if (slice->m_weightPredTable[l][0][0].bPresentFlag)470
+ if (slice->m_weightPredTable[l][0][0].wtPresent)471
bLuma = true;472
- if (slice->m_weightPredTable[l][0][1].bPresentFlag ||473
- slice->m_weightPredTable[l][0][2].bPresentFlag)474
+ if (slice->m_weightPredTable[l][0][1].wtPresent ||475
+ slice->m_weightPredTable[l][0][2].wtPresent)476
bChroma = true;477
}478
479
480
if (m_aborted)481
return -1;482
483
- finishFrameStats(outFrame, curEncoder, frameData, m_pocLast);484
+ if ((m_outputCount + 1) >= m_param->chunkStart)485
+ finishFrameStats(outFrame, curEncoder, frameData, m_pocLast);486
487
/* Write RateControl Frame level stats in multipass encodes */488
if (m_param->rc.bStatWrite)489
490
}491
else492
m_exportedPic = outFrame;493
-494
- m_numDelayedPic--;495
+ 496
+ m_outputCount++;497
+ if (m_param->chunkEnd == m_outputCount)498
+ m_numDelayedPic = 0;499
+ else 500
+ m_numDelayedPic--;501
502
ret = 1;503
}504
505
* curEncoder is guaranteed to be idle at this point */506
if (!pass)507
frameEnc = m_lookahead->getDecidedPicture();508
- if (frameEnc && !pass)509
+ if (frameEnc && !pass && (!m_param->chunkEnd || (m_encodedFrameNum < m_param->chunkEnd)))510
{511
if (m_param->analysisMultiPassRefine || m_param->analysisMultiPassDistortion)512
{513
- allocAnalysis2Pass(&frameEnc->m_analysis2Pass, frameEnc->m_lowres.sliceType);514
- frameEnc->m_analysis2Pass.poc = frameEnc->m_poc;515
+ uint32_t widthInCU = (m_param->sourceWidth + m_param->maxCUSize - 1) >> m_param->maxLog2CUSize;516
+ uint32_t heightInCU = (m_param->sourceHeight + m_param->maxCUSize - 1) >> m_param->maxLog2CUSize;517
+ frameEnc->m_analysisData.numCUsInFrame = widthInCU * heightInCU;518
+ frameEnc->m_analysisData.numPartitions = m_param->num4x4Partitions;519
+ x265_alloc_analysis_data(m_param, &frameEnc->m_analysisData);520
+ frameEnc->m_analysisData.poc = frameEnc->m_poc;521
if (m_param->rc.bStatRead)522
- readAnalysis2PassFile(&frameEnc->m_analysis2Pass, frameEnc->m_poc, frameEnc->m_lowres.sliceType);523
+ readAnalysisFile(&frameEnc->m_analysisData, frameEnc->m_poc, frameEnc->m_lowres.sliceType);524
}525
526
if (frameEnc->m_reconfigureRc && m_reconfigureRc)527
528
if (m_param->analysisLoad && m_param->bDisableLookahead)529
{530
frameEnc->m_dts = frameEnc->m_analysisData.lookahead.dts;531
+ frameEnc->m_reorderedPts = frameEnc->m_analysisData.lookahead.reorderedPts;532
if (m_rateControl->m_isVbv)533
{534
for (uint32_t index = 0; index < frameEnc->m_analysisData.numCuInHeight; index++)535
536
frameEnc->m_encData->m_slice->m_iNumRPSInSPS = m_sps.spsrpsNum;537
538
curEncoder->m_rce.encodeOrder = frameEnc->m_encodeOrder = m_encodedFrameNum++;539
+540
if (!m_param->analysisLoad || !m_param->bDisableLookahead)541
{542
if (m_bframeDelay)543
544
analysis->numCUsInFrame = numCUsInFrame;545
analysis->numCuInHeight = heightInCU;546
analysis->numPartitions = m_param->num4x4Partitions;547
- allocAnalysis(analysis);548
+ x265_alloc_analysis_data(m_param, analysis);549
}550
/* determine references, setup RPS, etc */551
m_dpb->prepareEncode(frameEnc);552
553
{554
const int picOrderCntLSB = slice->m_poc - slice->m_lastIDR;555
556
- frameStats->encoderOrder = m_outputCount++;557
+ frameStats->encoderOrder = m_outputCount;558
frameStats->sliceType = c;559
frameStats->poc = picOrderCntLSB;560
frameStats->qp = curEncData.m_avgQpAq;561
562
if (m_param->csvLogLevel >= 2)563
frameStats->ipCostRatio = curFrame->m_lowres.ipCostRatio;564
frameStats->bufferFill = m_rateControl->m_bufferFillActual;565
+ frameStats->bufferFillFinal = m_rateControl->m_bufferFillFinal;566
frameStats->frameLatency = inPoc - poc;567
if (m_param->rc.rateControlMode == X265_RC_CRF)568
frameStats->rateFactor = curEncData.m_rateFactor;569
570
#define ELAPSED_MSEC(start, end) (((double)(end) - (start)) / 1000)571
if (m_param->csvLogLevel >= 2)572
{573
+#if ENABLE_LIBVMAF574
+ frameStats->vmafFrameScore = curFrame->m_fencPic->m_vmafScore;575
+#endif576
frameStats->decideWaitTime = ELAPSED_MSEC(0, curEncoder->m_slicetypeWaitTime);577
frameStats->row0WaitTime = ELAPSED_MSEC(curEncoder->m_startCompressTime, curEncoder->m_row0WaitTime);578
frameStats->wallTime = ELAPSED_MSEC(curEncoder->m_row0WaitTime, curEncoder->m_endCompressTime);579
580
list.serialize(NAL_UNIT_SPS, bs);581
582
bs.resetBits();583
- sbacCoder.codePPS( m_pps, (m_param->maxSlices <= 1), m_iPPSQpMinus26);584
+ sbacCoder.codePPS(m_pps, (m_param->maxSlices <= 1), m_iPPSQpMinus26);585
bs.writeByteAlignment();586
list.serialize(NAL_UNIT_PPS, bs);587
588
+ if (m_param->bSingleSeiNal)589
+ bs.resetBits();590
+591
if (m_param->bEmitHDRSEI)592
{593
SEIContentLightLevel cllsei;594
cllsei.max_content_light_level = m_param->maxCLL;595
cllsei.max_pic_average_light_level = m_param->maxFALL;596
- bs.resetBits();597
- cllsei.write(bs, m_sps);598
- bs.writeByteAlignment();599
- list.serialize(NAL_UNIT_PREFIX_SEI, bs);600
+ cllsei.writeSEImessages(bs, m_sps, NAL_UNIT_PREFIX_SEI, list, m_param->bSingleSeiNal);601
602
if (m_param->masteringDisplayColorVolume)603
{604
SEIMasteringDisplayColorVolume mdsei;605
if (mdsei.parse(m_param->masteringDisplayColorVolume))606
- {607
- bs.resetBits();608
- mdsei.write(bs, m_sps);609
- bs.writeByteAlignment();610
- list.serialize(NAL_UNIT_PREFIX_SEI, bs);611
- }612
+ mdsei.writeSEImessages(bs, m_sps, NAL_UNIT_PREFIX_SEI, list, m_param->bSingleSeiNal);613
else614
x265_log(m_param, X265_LOG_WARNING, "unable to parse mastering display color volume info\n");615
}616
617
if (opts)618
{619
char *buffer = X265_MALLOC(char, strlen(opts) + strlen(PFX(version_str)) +620
- strlen(PFX(build_info_str)) + 200);621
+ strlen(PFX(build_info_str)) + 200);622
if (buffer)623
{624
sprintf(buffer, "x265 (build %d) - %s:%s - H.265/HEVC codec - "625
- "Copyright 2013-2018 (c) Multicoreware, Inc - "626
- "http://x265.org - options: %s",627
- X265_BUILD, PFX(version_str), PFX(build_info_str), opts);628
- 629
- bs.resetBits();630
+ "Copyright 2013-2018 (c) Multicoreware, Inc - "631
+ "http://x265.org - options: %s",632
+ X265_BUILD, PFX(version_str), PFX(build_info_str), opts);633
+634
SEIuserDataUnregistered idsei;635
idsei.m_userData = (uint8_t*)buffer;636
idsei.setSize((uint32_t)strlen(buffer));637
- idsei.write(bs, m_sps);638
- bs.writeByteAlignment();639
- list.serialize(NAL_UNIT_PREFIX_SEI, bs);640
+ idsei.writeSEImessages(bs, m_sps, NAL_UNIT_PREFIX_SEI, list, m_param->bSingleSeiNal);641
642
X265_FREE(buffer);643
}644
645
SEIActiveParameterSets sei;646
sei.m_selfContainedCvsFlag = true;647
sei.m_noParamSetUpdateFlag = true;648
-649
- bs.resetBits();650
- sei.write(bs, m_sps);651
- bs.writeByteAlignment();652
- list.serialize(NAL_UNIT_PREFIX_SEI, bs);653
+ sei.writeSEImessages(bs, m_sps, NAL_UNIT_PREFIX_SEI, list, m_param->bSingleSeiNal);654
}655
}656
657
658
vui.defaultDisplayWindow.bottomOffset = m_param->vui.defDispWinBottomOffset;659
vui.defaultDisplayWindow.leftOffset = m_param->vui.defDispWinLeftOffset;660
661
- vui.frameFieldInfoPresentFlag = !!m_param->interlaceMode;662
+ vui.frameFieldInfoPresentFlag = !!m_param->interlaceMode || (m_param->pictureStructure >= 0);663
vui.fieldSeqFlag = !!m_param->interlaceMode;664
665
vui.hrdParametersPresentFlag = m_param->bEmitHRDSEI;666
667
void Encoder::initPPS(PPS *pps)668
{669
bool bIsVbv = m_param->rc.vbvBufferSize > 0 && m_param->rc.vbvMaxBitrate > 0;670
+ bool bEnableDistOffset = m_param->analysisMultiPassDistortion && m_param->rc.bStatRead;671
672
if (!m_param->bLossless && (m_param->rc.aqMode || bIsVbv || m_param->bAQMotion))673
{674
675
pps->maxCuDQPDepth = g_log2Size[m_param->maxCUSize] - g_log2Size[m_param->rc.qgSize];676
X265_CHECK(pps->maxCuDQPDepth <= 3, "max CU DQP depth cannot be greater than 3\n");677
}678
+ else if (!m_param->bLossless && bEnableDistOffset)679
+ {680
+ pps->bUseDQP = true;681
+ pps->maxCuDQPDepth = 0;682
+ }683
else684
{685
pps->bUseDQP = false;686
687
{688
p->scaleFactor = 0;689
}690
- else if ((!p->analysisLoad && !p->analysisSave) || p->analysisReuseLevel < 10)691
+ else if ((!p->analysisLoad && !p->analysisSave) || (p->analysisReuseLevel > 6 && p->analysisReuseLevel != 10))692
{693
- x265_log(p, X265_LOG_WARNING, "Input scaling works with analysis load/save, analysis-reuse-level 10. Disabling scale-factor.\n");694
+ x265_log(p, X265_LOG_WARNING, "Input scaling works with analysis load/save and analysis-reuse-level 1-6 and 10. Disabling scale-factor.\n");695
p->scaleFactor = 0;696
}697
}698
699
if (p->intraRefine)700
{701
- if (!p->analysisLoad || p->analysisReuseLevel < 10 || !p->scaleFactor)702
+ if (!p->analysisLoad || p->analysisReuseLevel < 10)703
{704
- x265_log(p, X265_LOG_WARNING, "Intra refinement requires analysis load, analysis-reuse-level 10, scale factor. Disabling intra refine.\n");705
+ x265_log(p, X265_LOG_WARNING, "Intra refinement requires analysis load, analysis-reuse-level 10. Disabling intra refine.\n");706
p->intraRefine = 0;707
}708
}709
710
if (p->interRefine)711
{712
- if (!p->analysisLoad || p->analysisReuseLevel < 10 || !p->scaleFactor)713
+ if (!p->analysisLoad || p->analysisReuseLevel < 10)714
+ {715
+ x265_log(p, X265_LOG_WARNING, "Inter refinement requires analysis load, analysis-reuse-level 10. Disabling inter refine.\n");716
+ p->interRefine = 0;717
+ }718
+ }719
+720
+ if (p->bDynamicRefine)721
+ {722
+ if (!p->analysisLoad || p->analysisReuseLevel < 10)723
+ {724
+ x265_log(p, X265_LOG_WARNING, "Dynamic refinement requires analysis load, analysis-reuse-level 10. Disabling dynamic refine.\n");725
+ p->bDynamicRefine = 0;726
+ }727
+ if (p->interRefine)728
{729
- x265_log(p, X265_LOG_WARNING, "Inter refinement requires analysis load, analysis-reuse-level 10, scale factor. Disabling inter refine.\n");730
+ x265_log(p, X265_LOG_WARNING, "Inter refine cannot be used with dynamic refine. Disabling refine-inter.\n");731
p->interRefine = 0;732
}733
}734
+ if (p->scaleFactor && p->analysisLoad && !p->interRefine && !p->bDynamicRefine && p->analysisReuseLevel == 10)735
+ {736
+ x265_log(p, X265_LOG_WARNING, "Inter refinement 0 is not supported with scaling and analysis-reuse-level=10. Enabling refine-inter 1.\n");737
+ p->interRefine = 1;738
+ }739
740
- if (p->limitTU && p->interRefine)741
+ if (p->limitTU && (p->interRefine || p->bDynamicRefine))742
{743
x265_log(p, X265_LOG_WARNING, "Inter refinement does not support limitTU. Disabling limitTU.\n");744
p->limitTU = 0;745
746
747
if (p->mvRefine)748
{749
- if (!p->analysisLoad || p->analysisReuseLevel < 10 || !p->scaleFactor)750
+ if (!p->analysisLoad || p->analysisReuseLevel < 10)751
{752
- x265_log(p, X265_LOG_WARNING, "MV refinement requires analysis load, analysis-reuse-level 10, scale factor. Disabling MV refine.\n");753
+ x265_log(p, X265_LOG_WARNING, "MV refinement requires analysis load, analysis-reuse-level 10. Disabling MV refine.\n");754
p->mvRefine = 0;755
}756
else if (p->interRefine >= 2)757
758
p->bDistributeMotionEstimation = p->bDistributeModeAnalysis = 0;759
}760
761
- if (p->rc.bEnableGrain)762
- {763
- x265_log(p, X265_LOG_WARNING, "Rc Grain removes qp fluctuations caused by aq/cutree, Disabling aq,cu-tree\n");764
- p->rc.cuTree = 0;765
- p->rc.aqMode = 0;766
- }767
-768
if (p->bDistributeModeAnalysis && (p->limitReferences >> 1) && 1)769
{770
x265_log(p, X265_LOG_WARNING, "Limit reference options 2 and 3 are not supported with pmode. Disabling limit reference\n");771
772
p->radl = 0;773
x265_log(p, X265_LOG_WARNING, "Radl requires fixed gop-length (keyint == min-keyint). Disabling radl.\n");774
}775
-}776
-777
-void Encoder::allocAnalysis(x265_analysis_data* analysis)778
-{779
- X265_CHECK(analysis->sliceType, "invalid slice type\n");780
- analysis->interData = analysis->intraData = NULL;781
- if (m_param->bDisableLookahead && m_rateControl->m_isVbv)782
- {783
- CHECKED_MALLOC_ZERO(analysis->lookahead.intraSatdForVbv, uint32_t, analysis->numCuInHeight);784
- CHECKED_MALLOC_ZERO(analysis->lookahead.satdForVbv, uint32_t, analysis->numCuInHeight);785
- CHECKED_MALLOC_ZERO(analysis->lookahead.intraVbvCost, uint32_t, analysis->numCUsInFrame);786
- CHECKED_MALLOC_ZERO(analysis->lookahead.vbvCost, uint32_t, analysis->numCUsInFrame);787
- }788
- if (analysis->sliceType == X265_TYPE_IDR || analysis->sliceType == X265_TYPE_I)789
- {790
- if (m_param->analysisReuseLevel < 2)791
- return;792
-793
- analysis_intra_data *intraData = (analysis_intra_data*)analysis->intraData;794
- CHECKED_MALLOC_ZERO(intraData, analysis_intra_data, 1);795
- CHECKED_MALLOC(intraData->depth, uint8_t, analysis->numPartitions * analysis->numCUsInFrame);796
- CHECKED_MALLOC(intraData->modes, uint8_t, analysis->numPartitions * analysis->numCUsInFrame);797
- CHECKED_MALLOC(intraData->partSizes, char, analysis->numPartitions * analysis->numCUsInFrame);798
- CHECKED_MALLOC(intraData->chromaModes, uint8_t, analysis->numPartitions * analysis->numCUsInFrame);799
- analysis->intraData = intraData;800
- }801
- else802
- {803
- int numDir = analysis->sliceType == X265_TYPE_P ? 1 : 2;804
- uint32_t numPlanes = m_param->internalCsp == X265_CSP_I400 ? 1 : 3;805
- if (!(m_param->bMVType == AVC_INFO))806
- CHECKED_MALLOC_ZERO(analysis->wt, WeightParam, numPlanes * numDir);807
- if (m_param->analysisReuseLevel < 2)808
- return;809
-810
- analysis_inter_data *interData = (analysis_inter_data*)analysis->interData;811
- CHECKED_MALLOC_ZERO(interData, analysis_inter_data, 1);812
- CHECKED_MALLOC(interData->depth, uint8_t, analysis->numPartitions * analysis->numCUsInFrame);813
- CHECKED_MALLOC(interData->modes, uint8_t, analysis->numPartitions * analysis->numCUsInFrame);814
- if (m_param->analysisReuseLevel > 4)815
- {816
- CHECKED_MALLOC(interData->partSize, uint8_t, analysis->numPartitions * analysis->numCUsInFrame);817
- CHECKED_MALLOC(interData->mergeFlag, uint8_t, analysis->numPartitions * analysis->numCUsInFrame);818
- }819
-820
- if (m_param->analysisReuseLevel >= 7)821
- {822
- CHECKED_MALLOC(interData->interDir, uint8_t, analysis->numPartitions * analysis->numCUsInFrame);823
- CHECKED_MALLOC(interData->sadCost, int64_t, analysis->numPartitions * analysis->numCUsInFrame);824
- for (int dir = 0; dir < numDir; dir++)825
- {826
- CHECKED_MALLOC(interData->mvpIdx[dir], uint8_t, analysis->numPartitions * analysis->numCUsInFrame);827
- CHECKED_MALLOC(interData->refIdx[dir], int8_t, analysis->numPartitions * analysis->numCUsInFrame);828
- CHECKED_MALLOC(interData->mv[dir], MV, analysis->numPartitions * analysis->numCUsInFrame);829
- CHECKED_MALLOC_ZERO(analysis->modeFlag[dir], uint8_t, analysis->numPartitions * analysis->numCUsInFrame);830
- }831
- /* Allocate intra in inter */832
- if (analysis->sliceType == X265_TYPE_P || m_param->bIntraInBFrames)833
- {834
- analysis_intra_data *intraData = (analysis_intra_data*)analysis->intraData;835
- CHECKED_MALLOC_ZERO(intraData, analysis_intra_data, 1);836
- CHECKED_MALLOC(intraData->modes, uint8_t, analysis->numPartitions * analysis->numCUsInFrame);837
- CHECKED_MALLOC(intraData->chromaModes, uint8_t, analysis->numPartitions * analysis->numCUsInFrame);838
- analysis->intraData = intraData;839
- }840
- }841
- else842
- CHECKED_MALLOC_ZERO(interData->ref, int32_t, analysis->numCUsInFrame * X265_MAX_PRED_MODE_PER_CTU * numDir);843
-844
- analysis->interData = interData;845
- }846
- return;847
-848
-fail:849
- freeAnalysis(analysis);850
- m_aborted = true;851
-}852
-void Encoder::freeAnalysis(x265_analysis_data* analysis)853
-{854
- if (m_param->bDisableLookahead && m_rateControl->m_isVbv)855
- {856
- X265_FREE(analysis->lookahead.satdForVbv);857
- X265_FREE(analysis->lookahead.intraSatdForVbv);858
- X265_FREE(analysis->lookahead.vbvCost);859
- X265_FREE(analysis->lookahead.intraVbvCost);860
- }861
- /* Early exit freeing weights alone if level is 1 (when there is no analysis inter/intra) */862
- if (analysis->sliceType > X265_TYPE_I && analysis->wt && !(m_param->bMVType == AVC_INFO))863
- X265_FREE(analysis->wt);864
- if (m_param->analysisReuseLevel < 2)865
- return;866
867
- if (analysis->sliceType == X265_TYPE_IDR || analysis->sliceType == X265_TYPE_I)868
+ if ((p->chunkStart || p->chunkEnd) && p->bOpenGOP)869
{870
- if (analysis->intraData)871
- {872
- X265_FREE(((analysis_intra_data*)analysis->intraData)->depth);873
- X265_FREE(((analysis_intra_data*)analysis->intraData)->modes);874
- X265_FREE(((analysis_intra_data*)analysis->intraData)->partSizes);875
- X265_FREE(((analysis_intra_data*)analysis->intraData)->chromaModes);876
- X265_FREE(analysis->intraData);877
- analysis->intraData = NULL;878
- }879
+ p->chunkStart = p->chunkEnd = 0;880
+ x265_log(p, X265_LOG_WARNING, "Chunking requires closed gop structure. Disabling chunking.\n");881
}882
- else883
- {884
- if (analysis->intraData)885
- {886
- X265_FREE(((analysis_intra_data*)analysis->intraData)->modes);887
- X265_FREE(((analysis_intra_data*)analysis->intraData)->chromaModes);888
- X265_FREE(analysis->intraData);889
- analysis->intraData = NULL;890
- }891
- if (analysis->interData)892
- {893
- X265_FREE(((analysis_inter_data*)analysis->interData)->depth);894
- X265_FREE(((analysis_inter_data*)analysis->interData)->modes);895
- if (m_param->analysisReuseLevel > 4)896
- {897
- X265_FREE(((analysis_inter_data*)analysis->interData)->mergeFlag);898
- X265_FREE(((analysis_inter_data*)analysis->interData)->partSize);899
- }900
- if (m_param->analysisReuseLevel >= 7)901
- {902
- X265_FREE(((analysis_inter_data*)analysis->interData)->interDir);903
- X265_FREE(((analysis_inter_data*)analysis->interData)->sadCost);904
- int numDir = analysis->sliceType == X265_TYPE_P ? 1 : 2;905
- for (int dir = 0; dir < numDir; dir++)906
- {907
- X265_FREE(((analysis_inter_data*)analysis->interData)->mvpIdx[dir]);908
- X265_FREE(((analysis_inter_data*)analysis->interData)->refIdx[dir]);909
- X265_FREE(((analysis_inter_data*)analysis->interData)->mv[dir]);910
- if (analysis->modeFlag[dir] != NULL)911
- {912
- X265_FREE(analysis->modeFlag[dir]);913
- analysis->modeFlag[dir] = NULL;914
- }915
- }916
- }917
- else918
- X265_FREE(((analysis_inter_data*)analysis->interData)->ref);919
-920
- X265_FREE(analysis->interData);921
- analysis->interData = NULL;922
- }923
- }924
-}925
-926
-void Encoder::allocAnalysis2Pass(x265_analysis_2Pass* analysis, int sliceType)927
-{928
- analysis->analysisFramedata = NULL;929
- analysis2PassFrameData *analysisFrameData = (analysis2PassFrameData*)analysis->analysisFramedata;930
- uint32_t widthInCU = (m_param->sourceWidth + m_param->maxCUSize - 1) >> m_param->maxLog2CUSize;931
- uint32_t heightInCU = (m_param->sourceHeight + m_param->maxCUSize - 1) >> m_param->maxLog2CUSize;932
933
- uint32_t numCUsInFrame = widthInCU * heightInCU;934
- CHECKED_MALLOC_ZERO(analysisFrameData, analysis2PassFrameData, 1);935
- CHECKED_MALLOC_ZERO(analysisFrameData->depth, uint8_t, m_param->num4x4Partitions * numCUsInFrame);936
- CHECKED_MALLOC_ZERO(analysisFrameData->distortion, sse_t, m_param->num4x4Partitions * numCUsInFrame);937
- if (m_param->rc.bStatRead)938
- {939
- CHECKED_MALLOC_ZERO(analysisFrameData->ctuDistortion, sse_t, numCUsInFrame);940
- CHECKED_MALLOC_ZERO(analysisFrameData->scaledDistortion, double, numCUsInFrame);941
- CHECKED_MALLOC_ZERO(analysisFrameData->offset, double, numCUsInFrame);942
- CHECKED_MALLOC_ZERO(analysisFrameData->threshold, double, numCUsInFrame);943
- }944
- if (!IS_X265_TYPE_I(sliceType))945
+ if (p->chunkEnd < p->chunkStart)946
{947
- CHECKED_MALLOC_ZERO(analysisFrameData->m_mv[0], MV, m_param->num4x4Partitions * numCUsInFrame);948
- CHECKED_MALLOC_ZERO(analysisFrameData->m_mv[1], MV, m_param->num4x4Partitions * numCUsInFrame);949
- CHECKED_MALLOC_ZERO(analysisFrameData->mvpIdx[0], int, m_param->num4x4Partitions * numCUsInFrame);950
- CHECKED_MALLOC_ZERO(analysisFrameData->mvpIdx[1], int, m_param->num4x4Partitions * numCUsInFrame);951
- CHECKED_MALLOC_ZERO(analysisFrameData->ref[0], int32_t, m_param->num4x4Partitions * numCUsInFrame);952
- CHECKED_MALLOC_ZERO(analysisFrameData->ref[1], int32_t, m_param->num4x4Partitions * numCUsInFrame);953
- CHECKED_MALLOC(analysisFrameData->modes, uint8_t, m_param->num4x4Partitions * numCUsInFrame);954
+ p->chunkStart = p->chunkEnd = 0;955
+ x265_log(p, X265_LOG_WARNING, "chunk-end cannot be less than chunk-start. Disabling chunking.\n");956
}957
958
- analysis->analysisFramedata = analysisFrameData;959
-960
- return;961
-962
-fail:963
- freeAnalysis2Pass(analysis, sliceType);964
- m_aborted = true;965
-}966
-967
-void Encoder::freeAnalysis2Pass(x265_analysis_2Pass* analysis, int sliceType)968
-{969
- if (analysis->analysisFramedata)970
- {971
- X265_FREE(((analysis2PassFrameData*)analysis->analysisFramedata)->depth);972
- X265_FREE(((analysis2PassFrameData*)analysis->analysisFramedata)->distortion);973
- if (m_param->rc.bStatRead)974
- {975
- X265_FREE(((analysis2PassFrameData*)analysis->analysisFramedata)->ctuDistortion);976
- X265_FREE(((analysis2PassFrameData*)analysis->analysisFramedata)->scaledDistortion);977
- X265_FREE(((analysis2PassFrameData*)analysis->analysisFramedata)->offset);978
- X265_FREE(((analysis2PassFrameData*)analysis->analysisFramedata)->threshold);979
- }980
- if (!IS_X265_TYPE_I(sliceType))981
- {982
- X265_FREE(((analysis2PassFrameData*)analysis->analysisFramedata)->m_mv[0]);983
- X265_FREE(((analysis2PassFrameData*)analysis->analysisFramedata)->m_mv[1]);984
- X265_FREE(((analysis2PassFrameData*)analysis->analysisFramedata)->mvpIdx[0]);985
- X265_FREE(((analysis2PassFrameData*)analysis->analysisFramedata)->mvpIdx[1]);986
- X265_FREE(((analysis2PassFrameData*)analysis->analysisFramedata)->ref[0]);987
- X265_FREE(((analysis2PassFrameData*)analysis->analysisFramedata)->ref[1]);988
- X265_FREE(((analysis2PassFrameData*)analysis->analysisFramedata)->modes);989
- }990
- X265_FREE(analysis->analysisFramedata);991
- }992
}993
994
-void Encoder::readAnalysisFile(x265_analysis_data* analysis, int curPoc, const x265_picture* picIn)995
+void Encoder::readAnalysisFile(x265_analysis_data* analysis, int curPoc, const x265_picture* picIn, int paramBytes)996
{997
-998
#define X265_FREAD(val, size, readSize, fileOffset, src)\999
if (!m_param->bUseAnalysisFile)\1000
- {\1001
+ {\1002
memcpy(val, src, (size * readSize));\1003
- }\1004
- else if (fread(val, size, readSize, fileOffset) != readSize)\1005
+ }\1006
+ else if (fread(val, size, readSize, fileOffset) != readSize)\1007
{\1008
x265_log(NULL, X265_LOG_ERROR, "Error reading analysis data\n");\1009
- freeAnalysis(analysis);\1010
+ x265_free_analysis_data(m_param, analysis);\1011
m_aborted = true;\1012
return;\1013
}\1014
1015
static uint64_t totalConsumedBytes = 0;1016
uint32_t depthBytes = 0;1017
if (m_param->bUseAnalysisFile)1018
- fseeko(m_analysisFileIn, totalConsumedBytes, SEEK_SET);1019
+ fseeko(m_analysisFileIn, totalConsumedBytes + paramBytes, SEEK_SET);1020
const x265_analysis_data *picData = &(picIn->analysisData);1021
- analysis_intra_data *intraPic = (analysis_intra_data *)picData->intraData;1022
- analysis_inter_data *interPic = (analysis_inter_data *)picData->interData;1023
+ x265_analysis_intra_data *intraPic = picData->intraData;1024
+ x265_analysis_inter_data *interPic = picData->interData;1025
1026
int poc; uint32_t frameRecordSize;1027
X265_FREAD(&frameRecordSize, sizeof(uint32_t), 1, m_analysisFileIn, &(picData->frameRecordSize));1028
1029
while (poc != curPoc && !feof(m_analysisFileIn))1030
{1031
currentOffset += frameRecordSize;1032
- fseeko(m_analysisFileIn, currentOffset, SEEK_SET);1033
+ fseeko(m_analysisFileIn, currentOffset + paramBytes, SEEK_SET);1034
X265_FREAD(&frameRecordSize, sizeof(uint32_t), 1, m_analysisFileIn, &(picData->frameRecordSize));1035
X265_FREAD(&depthBytes, sizeof(uint32_t), 1, m_analysisFileIn, &(picData->depthBytes));1036
X265_FREAD(&poc, sizeof(int), 1, m_analysisFileIn, &(picData->poc));1037
1038
if (poc != curPoc || feof(m_analysisFileIn))1039
{1040
x265_log(NULL, X265_LOG_WARNING, "Error reading analysis data: Cannot find POC %d\n", curPoc);1041
- freeAnalysis(analysis);1042
+ x265_free_analysis_data(m_param, analysis);1043
return;1044
}1045
}1046
1047
if (m_param->scaleFactor)1048
analysis->numPartitions *= factor;1049
/* Memory is allocated for inter and intra analysis data based on the slicetype */1050
- allocAnalysis(analysis);1051
+ x265_alloc_analysis_data(m_param, analysis);1052
if (m_param->bDisableLookahead && m_rateControl->m_isVbv)1053
{1054
+ size_t vbvCount = m_param->lookaheadDepth + m_param->bframes + 2;1055
X265_FREAD(analysis->lookahead.intraVbvCost, sizeof(uint32_t), analysis->numCUsInFrame, m_analysisFileIn, picData->lookahead.intraVbvCost);1056
X265_FREAD(analysis->lookahead.vbvCost, sizeof(uint32_t), analysis->numCUsInFrame, m_analysisFileIn, picData->lookahead.vbvCost);1057
X265_FREAD(analysis->lookahead.satdForVbv, sizeof(uint32_t), analysis->numCuInHeight, m_analysisFileIn, picData->lookahead.satdForVbv);1058
X265_FREAD(analysis->lookahead.intraSatdForVbv, sizeof(uint32_t), analysis->numCuInHeight, m_analysisFileIn, picData->lookahead.intraSatdForVbv);1059
+ X265_FREAD(analysis->lookahead.plannedSatd, sizeof(int64_t), vbvCount, m_analysisFileIn, picData->lookahead.plannedSatd);1060
+1061
+ if (m_param->scaleFactor)1062
+ {1063
+ for (uint64_t index = 0; index < vbvCount; index++)1064
+ analysis->lookahead.plannedSatd[index] *= factor;1065
+1066
+ for (uint32_t i = 0; i < analysis->numCuInHeight; i++)1067
+ {1068
+ analysis->lookahead.satdForVbv[i] *= factor;1069
+ analysis->lookahead.intraSatdForVbv[i] *= factor;1070
+ }1071
+ for (uint32_t i = 0; i < analysis->numCUsInFrame; i++)1072
+ {1073
+ analysis->lookahead.vbvCost[i] *= factor;1074
+ analysis->lookahead.intraVbvCost[i] *= factor;1075
+ }1076
+ }1077
}1078
if (analysis->sliceType == X265_TYPE_IDR || analysis->sliceType == X265_TYPE_I)1079
{1080
1081
if (partSizes[d] == SIZE_NxN)1082
partSizes[d] = SIZE_2Nx2N;1083
}1084
- memset(&((analysis_intra_data *)analysis->intraData)->depth[count], depthBuf[d], bytes);1085
- memset(&((analysis_intra_data *)analysis->intraData)->chromaModes[count], modeBuf[d], bytes);1086
- memset(&((analysis_intra_data *)analysis->intraData)->partSizes[count], partSizes[d], bytes);1087
+ memset(&(analysis->intraData)->depth[count], depthBuf[d], bytes);1088
+ memset(&(analysis->intraData)->chromaModes[count], modeBuf[d], bytes);1089
+ memset(&(analysis->intraData)->partSizes[count], partSizes[d], bytes);1090
count += bytes;1091
}1092
1093
if (!m_param->scaleFactor)1094
{1095
- X265_FREAD(((analysis_intra_data *)analysis->intraData)->modes, sizeof(uint8_t), analysis->numCUsInFrame * analysis->numPartitions, m_analysisFileIn, intraPic->modes);1096
+ X265_FREAD((analysis->intraData)->modes, sizeof(uint8_t), analysis->numCUsInFrame * analysis->numPartitions, m_analysisFileIn, intraPic->modes);1097
}1098
else1099
{1100
uint8_t *tempLumaBuf = X265_MALLOC(uint8_t, analysis->numCUsInFrame * scaledNumPartition);1101
X265_FREAD(tempLumaBuf, sizeof(uint8_t), analysis->numCUsInFrame * scaledNumPartition, m_analysisFileIn, intraPic->modes);1102
for (uint32_t ctu32Idx = 0, cnt = 0; ctu32Idx < analysis->numCUsInFrame * scaledNumPartition; ctu32Idx++, cnt += factor)1103
- memset(&((analysis_intra_data *)analysis->intraData)->modes[cnt], tempLumaBuf[ctu32Idx], factor);1104
+ memset(&(analysis->intraData)->modes[cnt], tempLumaBuf[ctu32Idx], factor);1105
X265_FREE(tempLumaBuf);1106
}1107
X265_FREE(tempBuf);1108
1109
{1110
int bytes = analysis->numPartitions >> (depthBuf[d] * 2);1111
if (m_param->scaleFactor && modeBuf[d] == MODE_INTRA && depthBuf[d] == 0)1112
- depthBuf[d] = 1;1113
- memset(&((analysis_inter_data *)analysis->interData)->depth[count], depthBuf[d], bytes);1114
- memset(&((analysis_inter_data *)analysis->interData)->modes[count], modeBuf[d], bytes);1115
+ depthBuf[d] = 1;1116
+ memset(&(analysis->interData)->depth[count], depthBuf[d], bytes);1117
+ memset(&(analysis->interData)->modes[count], modeBuf[d], bytes);1118
if (m_param->analysisReuseLevel > 4)1119
{1120
if (m_param->scaleFactor && modeBuf[d] == MODE_INTRA && partSize[d] == SIZE_NxN)1121
- partSize[d] = SIZE_2Nx2N;1122
- memset(&((analysis_inter_data *)analysis->interData)->partSize[count], partSize[d], bytes);1123
+ partSize[d] = SIZE_2Nx2N;1124
+ memset(&(analysis->interData)->partSize[count], partSize[d], bytes);1125
int numPU = (modeBuf[d] == MODE_INTRA) ? 1 : nbPartsTable[(int)partSize[d]];1126
for (int pu = 0; pu < numPU; pu++)1127
{1128
if (pu) d++;1129
- ((analysis_inter_data *)analysis->interData)->mergeFlag[count + pu] = mergeFlag[d];1130
+ (analysis->interData)->mergeFlag[count + pu] = mergeFlag[d];1131
if (m_param->analysisReuseLevel == 10)1132
{1133
- ((analysis_inter_data *)analysis->interData)->interDir[count + pu] = interDir[d];1134
+ (analysis->interData)->interDir[count + pu] = interDir[d];1135
for (uint32_t i = 0; i < numDir; i++)1136
{1137
- ((analysis_inter_data *)analysis->interData)->mvpIdx[i][count + pu] = mvpIdx[i][d];1138
- ((analysis_inter_data *)analysis->interData)->refIdx[i][count + pu] = refIdx[i][d];1139
+ (analysis->interData)->mvpIdx[i][count + pu] = mvpIdx[i][d];1140
+ (analysis->interData)->refIdx[i][count + pu] = refIdx[i][d];1141
if (m_param->scaleFactor)1142
{1143
mv[i][d].x *= (int16_t)m_param->scaleFactor;1144
mv[i][d].y *= (int16_t)m_param->scaleFactor;1145
}1146
- memcpy(&((analysis_inter_data *)analysis->interData)->mv[i][count + pu], &mv[i][d], sizeof(MV));1147
+ memcpy(&(analysis->interData)->mv[i][count + pu], &mv[i][d], sizeof(MV));1148
}1149
}1150
}1151
if (m_param->analysisReuseLevel == 10 && bIntraInInter)1152
- memset(&((analysis_intra_data *)analysis->intraData)->chromaModes[count], chromaDir[d], bytes);1153
+ memset(&(analysis->intraData)->chromaModes[count], chromaDir[d], bytes);1154
}1155
count += bytes;1156
}1157
1158
{1159
if (!m_param->scaleFactor)1160
{1161
- X265_FREAD(((analysis_intra_data *)analysis->intraData)->modes, sizeof(uint8_t), analysis->numCUsInFrame * analysis->numPartitions, m_analysisFileIn, intraPic->modes);1162
+ X265_FREAD((analysis->intraData)->modes, sizeof(uint8_t), analysis->numCUsInFrame * analysis->numPartitions, m_analysisFileIn, intraPic->modes);1163
}1164
else1165
{1166
uint8_t *tempLumaBuf = X265_MALLOC(uint8_t, analysis->numCUsInFrame * scaledNumPartition);1167
X265_FREAD(tempLumaBuf, sizeof(uint8_t), analysis->numCUsInFrame * scaledNumPartition, m_analysisFileIn, intraPic->modes);1168
for (uint32_t ctu32Idx = 0, cnt = 0; ctu32Idx < analysis->numCUsInFrame * scaledNumPartition; ctu32Idx++, cnt += factor)1169
- memset(&((analysis_intra_data *)analysis->intraData)->modes[cnt], tempLumaBuf[ctu32Idx], factor);1170
+ memset(&(analysis->intraData)->modes[cnt], tempLumaBuf[ctu32Idx], factor);1171
X265_FREE(tempLumaBuf);1172
}1173
}1174
}1175
else1176
- X265_FREAD(((analysis_inter_data *)analysis->interData)->ref, sizeof(int32_t), analysis->numCUsInFrame * X265_MAX_PRED_MODE_PER_CTU * numDir, m_analysisFileIn, interPic->ref);1177
+ X265_FREAD((analysis->interData)->ref, sizeof(int32_t), analysis->numCUsInFrame * X265_MAX_PRED_MODE_PER_CTU * numDir, m_analysisFileIn, interPic->ref);1178
1179
consumedBytes += frameRecordSize;1180
if (numDir == 1)1181
1182
#undef X265_FREAD1183
}1184
1185
-void Encoder::readAnalysis2PassFile(x265_analysis_2Pass* analysis2Pass, int curPoc, int sliceType)1186
+void Encoder::readAnalysisFile(x265_analysis_data* analysis, int curPoc, const x265_picture* picIn, int paramBytes, cuLocation cuLoc)1187
+{1188
+#define X265_FREAD(val, size, readSize, fileOffset, src)\1189
+ if (!m_param->bUseAnalysisFile)\1190
+ {\1191
+ memcpy(val, src, (size * readSize));\1192
+ }\1193
+ else if (fread(val, size, readSize, fileOffset) != readSize)\1194
+ {\1195
+ x265_log(NULL, X265_LOG_ERROR, "Error reading analysis data\n");\1196
+ x265_free_analysis_data(m_param, analysis);\1197
+ m_aborted = true;\1198
+ return;\1199
+ }\1200
+1201
+ static uint64_t consumedBytes = 0;1202
+ static uint64_t totalConsumedBytes = 0;1203
+ uint32_t depthBytes = 0;1204
+ if (m_param->bUseAnalysisFile)1205
+ fseeko(m_analysisFileIn, totalConsumedBytes + paramBytes, SEEK_SET);1206
+1207
+ const x265_analysis_data *picData = &(picIn->analysisData);1208
+ x265_analysis_intra_data *intraPic = picData->intraData;1209
+ x265_analysis_inter_data *interPic = picData->interData;1210
+1211
+ int poc; uint32_t frameRecordSize;1212
+ X265_FREAD(&frameRecordSize, sizeof(uint32_t), 1, m_analysisFileIn, &(picData->frameRecordSize));1213
+ X265_FREAD(&depthBytes, sizeof(uint32_t), 1, m_analysisFileIn, &(picData->depthBytes));1214
+ X265_FREAD(&poc, sizeof(int), 1, m_analysisFileIn, &(picData->poc));1215
+1216
+ if (m_param->bUseAnalysisFile)1217
+ {1218
+ uint64_t currentOffset = totalConsumedBytes;1219
+1220
+ /* Seeking to the right frame Record */1221
+ while (poc != curPoc && !feof(m_analysisFileIn))1222
+ {1223
+ currentOffset += frameRecordSize;1224
+ fseeko(m_analysisFileIn, currentOffset + paramBytes, SEEK_SET);1225
+ X265_FREAD(&frameRecordSize, sizeof(uint32_t), 1, m_analysisFileIn, &(picData->frameRecordSize));1226
+ X265_FREAD(&depthBytes, sizeof(uint32_t), 1, m_analysisFileIn, &(picData->depthBytes));1227
+ X265_FREAD(&poc, sizeof(int), 1, m_analysisFileIn, &(picData->poc));1228
+ }1229
+ if (poc != curPoc || feof(m_analysisFileIn))1230
+ {1231
+ x265_log(NULL, X265_LOG_WARNING, "Error reading analysis data: Cannot find POC %d\n", curPoc);1232
+ x265_free_analysis_data(m_param, analysis);1233
+ return;1234
+ }1235
+ }1236
+1237
+ /* Now arrived at the right frame, read the record */1238
+ analysis->poc = poc;1239
+ analysis->frameRecordSize = frameRecordSize;1240
+ X265_FREAD(&analysis->sliceType, sizeof(int), 1, m_analysisFileIn, &(picData->sliceType));1241
+ X265_FREAD(&analysis->bScenecut, sizeof(int), 1, m_analysisFileIn, &(picData->bScenecut));1242
+ X265_FREAD(&analysis->satdCost, sizeof(int64_t), 1, m_analysisFileIn, &(picData->satdCost));1243
+ X265_FREAD(&analysis->numCUsInFrame, sizeof(int), 1, m_analysisFileIn, &(picData->numCUsInFrame));1244
+ X265_FREAD(&analysis->numPartitions, sizeof(int), 1, m_analysisFileIn, &(picData->numPartitions));1245
+ if (m_param->bDisableLookahead)1246
+ {1247
+ X265_FREAD(&analysis->numCuInHeight, sizeof(uint32_t), 1, m_analysisFileIn, &(picData->numCuInHeight));1248
+ X265_FREAD(&analysis->lookahead, sizeof(x265_lookahead_data), 1, m_analysisFileIn, &(picData->lookahead));1249
+ }1250
+ int scaledNumPartition = analysis->numPartitions;1251
+ int factor = 1 << m_param->scaleFactor;1252
+1253
+ int numPartitions = analysis->numPartitions;1254
+ int numCUsInFrame = analysis->numCUsInFrame;1255
+ int numCuInHeight = analysis->numCuInHeight;1256
+ /* Allocate memory for scaled resoultion's numPartitions and numCUsInFrame*/1257
+ analysis->numPartitions = m_param->num4x4Partitions;1258
+ analysis->numCUsInFrame = cuLoc.heightInCU * cuLoc.widthInCU;1259
+ analysis->numCuInHeight = cuLoc.heightInCU;1260
+1261
+ /* Memory is allocated for inter and intra analysis data based on the slicetype */1262
+ x265_alloc_analysis_data(m_param, analysis);1263
+1264
+ analysis->numPartitions = numPartitions * factor;1265
+ analysis->numCUsInFrame = numCUsInFrame;1266
+ analysis->numCuInHeight = numCuInHeight;1267
+ if (m_param->bDisableLookahead && m_rateControl->m_isVbv)1268
+ {1269
+ uint32_t width = analysis->numCUsInFrame / analysis->numCuInHeight;1270
+ bool skipLastRow = (analysis->numCuInHeight * 2) > cuLoc.heightInCU;1271
+ bool skipLastCol = (width * 2) > cuLoc.widthInCU;1272
+ uint32_t *intraVbvCostBuf = NULL, *vbvCostBuf = NULL, *satdForVbvBuf = NULL, *intraSatdForVbvBuf = NULL;1273
+ intraVbvCostBuf = X265_MALLOC(uint32_t, analysis->numCUsInFrame);1274
+ vbvCostBuf = X265_MALLOC(uint32_t, analysis->numCUsInFrame);1275
+ satdForVbvBuf = X265_MALLOC(uint32_t, analysis->numCuInHeight);1276
+ intraSatdForVbvBuf = X265_MALLOC(uint32_t, analysis->numCuInHeight);1277
+1278
+ X265_FREAD(intraVbvCostBuf, sizeof(uint32_t), analysis->numCUsInFrame, m_analysisFileIn, picData->lookahead.intraVbvCost);1279
+ X265_FREAD(vbvCostBuf, sizeof(uint32_t), analysis->numCUsInFrame, m_analysisFileIn, picData->lookahead.vbvCost);1280
+ X265_FREAD(satdForVbvBuf, sizeof(uint32_t), analysis->numCuInHeight, m_analysisFileIn, picData->lookahead.satdForVbv);1281
+ X265_FREAD(intraSatdForVbvBuf, sizeof(uint32_t), analysis->numCuInHeight, m_analysisFileIn, picData->lookahead.intraSatdForVbv);1282
+1283
+ int k = 0;1284
+ for (uint32_t i = 0; i < analysis->numCuInHeight; i++)1285
+ {1286
+ analysis->lookahead.satdForVbv[m_param->scaleFactor * i] = satdForVbvBuf[i] * m_param->scaleFactor;1287
+ analysis->lookahead.intraSatdForVbv[m_param->scaleFactor * i] = intraSatdForVbvBuf[i] * m_param->scaleFactor;1288
+ if (!(i == (analysis->numCuInHeight - 1) && skipLastRow))1289
+ {1290
+ analysis->lookahead.satdForVbv[(m_param->scaleFactor * i) + 1] = satdForVbvBuf[i] * m_param->scaleFactor;1291
+ analysis->lookahead.intraSatdForVbv[(m_param->scaleFactor * i) + 1] = intraSatdForVbvBuf[i] * m_param->scaleFactor;1292
+ }1293
+1294
+ for (uint32_t j = 0; j < width; j++, k++)1295
+ {1296
+ analysis->lookahead.vbvCost[(i * m_param->scaleFactor * cuLoc.widthInCU) + (j * m_param->scaleFactor)] = vbvCostBuf[k];1297
+ analysis->lookahead.intraVbvCost[(i * m_param->scaleFactor * cuLoc.widthInCU) + (j * m_param->scaleFactor)] = intraVbvCostBuf[k];1298
+1299
+ if (!(j == (width - 1) && skipLastCol))1300
+ {1301
+ analysis->lookahead.vbvCost[(i * m_param->scaleFactor * cuLoc.widthInCU) + (j * m_param->scaleFactor) + 1] = vbvCostBuf[k];1302
+ analysis->lookahead.intraVbvCost[(i * m_param->scaleFactor * cuLoc.widthInCU) + (j * m_param->scaleFactor) + 1] = intraVbvCostBuf[k];1303
+ }1304
+ if (!(i == (analysis->numCuInHeight - 1) && skipLastRow))1305
+ {1306
+ analysis->lookahead.vbvCost[(i * m_param->scaleFactor * cuLoc.widthInCU) + cuLoc.widthInCU + (j * m_param->scaleFactor)] = vbvCostBuf[k];1307
+ analysis->lookahead.intraVbvCost[(i * m_param->scaleFactor * cuLoc.widthInCU) + cuLoc.widthInCU + (j * m_param->scaleFactor)] = intraVbvCostBuf[k];1308
+ if (!(j == (width - 1) && skipLastCol))1309
+ {1310
+ analysis->lookahead.vbvCost[(i * m_param->scaleFactor * cuLoc.widthInCU) + cuLoc.widthInCU + (j * m_param->scaleFactor) + 1] = vbvCostBuf[k];1311
+ analysis->lookahead.intraVbvCost[(i * m_param->scaleFactor * cuLoc.widthInCU) + cuLoc.widthInCU + (j * m_param->scaleFactor) + 1] = intraVbvCostBuf[k];1312
+ }1313
+ }1314
+ }1315
+ }1316
+ X265_FREE(satdForVbvBuf);1317
+ X265_FREE(intraSatdForVbvBuf);1318
+ X265_FREE(intraVbvCostBuf);1319
+ X265_FREE(vbvCostBuf);1320
+ }1321
+1322
+ if (analysis->sliceType == X265_TYPE_IDR || analysis->sliceType == X265_TYPE_I)1323
+ {1324
+ if (m_param->analysisReuseLevel < 2)1325
+ return;1326
+1327
+ uint8_t *tempBuf = NULL, *depthBuf = NULL, *modeBuf = NULL, *partSizes = NULL;1328
+1329
+ tempBuf = X265_MALLOC(uint8_t, depthBytes * 3);1330
+ depthBuf = tempBuf;1331
+ modeBuf = tempBuf + depthBytes;1332
+ partSizes = tempBuf + 2 * depthBytes;1333
+1334
+ X265_FREAD(depthBuf, sizeof(uint8_t), depthBytes, m_analysisFileIn, intraPic->depth);1335
+ X265_FREAD(modeBuf, sizeof(uint8_t), depthBytes, m_analysisFileIn, intraPic->chromaModes);1336
+ X265_FREAD(partSizes, sizeof(uint8_t), depthBytes, m_analysisFileIn, intraPic->partSizes);1337
+1338
+ uint32_t count = 0;1339
+ for (uint32_t d = 0; d < depthBytes; d++)1340
+ {1341
+ int bytes = analysis->numPartitions >> (depthBuf[d] * 2);1342
+ int numCTUCopied = 1;1343
+ if (!depthBuf[d]) //copy data of one 64x64 to four scaled 64x64 CTUs.1344
+ {1345
+ bytes /= 4;1346
+ numCTUCopied = 4;1347
+ }1348
+ if (partSizes[d] == SIZE_NxN)1349
+ partSizes[d] = SIZE_2Nx2N;1350
+ if ((depthBuf[d] > 1 && m_param->maxCUSize == 64) || (depthBuf[d] && m_param->maxCUSize != 64))1351
+ depthBuf[d]--;1352
+1353
+ for (int numCTU = 0; numCTU < numCTUCopied; numCTU++)1354
+ {1355
+ memset(&(analysis->intraData)->depth[count], depthBuf[d], bytes);1356
+ memset(&(analysis->intraData)->chromaModes[count], modeBuf[d], bytes);1357
+ memset(&(analysis->intraData)->partSizes[count], partSizes[d], bytes);1358
+ count += bytes;1359
+ d += getCUIndex(&cuLoc, &count, bytes, 1);1360
+ }1361
+ }1362
+1363
+ cuLoc.evenRowIndex = 0;1364
+ cuLoc.oddRowIndex = m_param->num4x4Partitions * cuLoc.widthInCU;1365
+ cuLoc.switchCondition = 0;1366
+ uint8_t *tempLumaBuf = X265_MALLOC(uint8_t, analysis->numCUsInFrame * scaledNumPartition);1367
+ X265_FREAD(tempLumaBuf, sizeof(uint8_t), analysis->numCUsInFrame * scaledNumPartition, m_analysisFileIn, intraPic->modes);1368
+ uint32_t cnt = 0;1369
+ for (uint32_t ctu32Idx = 0; ctu32Idx < analysis->numCUsInFrame * scaledNumPartition; ctu32Idx++)1370
+ {1371
+ memset(&(analysis->intraData)->modes[cnt], tempLumaBuf[ctu32Idx], factor);1372
+ cnt += factor;1373
+ ctu32Idx += getCUIndex(&cuLoc, &cnt, factor, 0);1374
+ }1375
+ X265_FREE(tempLumaBuf);1376
+ X265_FREE(tempBuf);1377
+ consumedBytes += frameRecordSize;1378
+ }1379
+1380
+ else1381
+ {1382
+ uint32_t numDir = analysis->sliceType == X265_TYPE_P ? 1 : 2;1383
+ uint32_t numPlanes = m_param->internalCsp == X265_CSP_I400 ? 1 : 3;1384
+ X265_FREAD((WeightParam*)analysis->wt, sizeof(WeightParam), numPlanes * numDir, m_analysisFileIn, (picIn->analysisData.wt));1385
+ if (m_param->analysisReuseLevel < 2)1386
+ return;1387
+1388
+ uint8_t *tempBuf = NULL, *depthBuf = NULL, *modeBuf = NULL, *partSize = NULL, *mergeFlag = NULL;1389
+ uint8_t *interDir = NULL, *chromaDir = NULL, *mvpIdx[2];1390
+ MV* mv[2];1391
+ int8_t* refIdx[2];1392
+1393
+ int numBuf = m_param->analysisReuseLevel > 4 ? 4 : 2;1394
+ bool bIntraInInter = false;1395
+ if (m_param->analysisReuseLevel == 10)1396
+ {1397
+ numBuf++;1398
+ bIntraInInter = (analysis->sliceType == X265_TYPE_P || m_param->bIntraInBFrames);1399
+ if (bIntraInInter) numBuf++;1400
+ }1401
+1402
+ tempBuf = X265_MALLOC(uint8_t, depthBytes * numBuf);1403
+ depthBuf = tempBuf;1404
+ modeBuf = tempBuf + depthBytes;1405
+1406
+ X265_FREAD(depthBuf, sizeof(uint8_t), depthBytes, m_analysisFileIn, interPic->depth);1407
+ X265_FREAD(modeBuf, sizeof(uint8_t), depthBytes, m_analysisFileIn, interPic->modes);1408
+ if (m_param->analysisReuseLevel > 4)1409
+ {1410
+ partSize = modeBuf + depthBytes;1411
+ mergeFlag = partSize + depthBytes;1412
+ X265_FREAD(partSize, sizeof(uint8_t), depthBytes, m_analysisFileIn, interPic->partSize);1413
+ X265_FREAD(mergeFlag, sizeof(uint8_t), depthBytes, m_analysisFileIn, interPic->mergeFlag);1414
+ if (m_param->analysisReuseLevel == 10)1415
+ {1416
+ interDir = mergeFlag + depthBytes;1417
+ X265_FREAD(interDir, sizeof(uint8_t), depthBytes, m_analysisFileIn, interPic->interDir);1418
+ if (bIntraInInter)1419
+ {1420
+ chromaDir = interDir + depthBytes;1421
+ X265_FREAD(chromaDir, sizeof(uint8_t), depthBytes, m_analysisFileIn, intraPic->chromaModes);1422
+ }1423
+ for (uint32_t i = 0; i < numDir; i++)1424
+ {1425
+ mvpIdx[i] = X265_MALLOC(uint8_t, depthBytes);1426
+ refIdx[i] = X265_MALLOC(int8_t, depthBytes);1427
+ mv[i] = X265_MALLOC(MV, depthBytes);1428
+ X265_FREAD(mvpIdx[i], sizeof(uint8_t), depthBytes, m_analysisFileIn, interPic->mvpIdx[i]);1429
+ X265_FREAD(refIdx[i], sizeof(int8_t), depthBytes, m_analysisFileIn, interPic->refIdx[i]);1430
+ X265_FREAD(mv[i], sizeof(MV), depthBytes, m_analysisFileIn, interPic->mv[i]);1431
+ }1432
+ }1433
+ }1434
+1435
+ uint32_t count = 0;1436
+ cuLoc.switchCondition = 0;1437
+ for (uint32_t d = 0; d < depthBytes; d++)1438
+ {1439
+ int bytes = analysis->numPartitions >> (depthBuf[d] * 2);1440
+ bool isScaledMaxCUSize = false;1441
+ int numCTUCopied = 1;1442
+ int writeDepth = depthBuf[d];1443
+ if (!depthBuf[d]) //copy data of one 64x64 to four scaled 64x64 CTUs.1444
+ {1445
+ isScaledMaxCUSize = true;1446
+ bytes /= 4;1447
+ numCTUCopied = 4;1448
+ }1449
+ if ((modeBuf[d] != MODE_INTRA && depthBuf[d] != 0) || (modeBuf[d] == MODE_INTRA && depthBuf[d] > 1))1450
+ writeDepth--;1451
+1452
+ for (int numCTU = 0; numCTU < numCTUCopied; numCTU++)1453
+ {1454
+ memset(&(analysis->interData)->depth[count], writeDepth, bytes);1455
+ memset(&(analysis->interData)->modes[count], modeBuf[d], bytes);1456
+ if (m_param->analysisReuseLevel == 10 && bIntraInInter)1457
+ memset(&(analysis->intraData)->chromaModes[count], chromaDir[d], bytes);1458
+1459
+ if (m_param->analysisReuseLevel > 4)1460
+ {1461
+ puOrientation puOrient;1462
+ puOrient.init();1463
+ if (modeBuf[d] == MODE_INTRA && partSize[d] == SIZE_NxN)1464
+ partSize[d] = SIZE_2Nx2N;1465
+ int partitionSize = partSize[d];1466
+ if (isScaledMaxCUSize && partSize[d] != SIZE_2Nx2N)1467
+ partitionSize = getPuShape(&puOrient, partSize[d], numCTU);1468
+ memset(&(analysis->interData)->partSize[count], partitionSize, bytes);1469
+ int numPU = (modeBuf[d] == MODE_INTRA) ? 1 : nbPartsTable[(int)partSize[d]];1470
+ for (int pu = 0; pu < numPU; pu++)1471
+ {1472
+ if (!isScaledMaxCUSize && pu)1473
+ d++;1474
+ int restoreD = d;1475
+ /* Adjust d value when the current CTU takes data from 2nd PU */1476
+ if (puOrient.isRect || (puOrient.isAmp && partitionSize == SIZE_2Nx2N))1477
+ {1478
+ if ((numCTU > 1 && !puOrient.isVert) || ((numCTU % 2 == 1) && puOrient.isVert))1479
+ d++;1480
+ }1481
+ if (puOrient.isAmp && pu)1482
+ d++;1483
+1484
+ (analysis->interData)->mergeFlag[count + pu] = mergeFlag[d];1485
+ if (m_param->analysisReuseLevel == 10)1486
+ {1487
+ (analysis->interData)->interDir[count + pu] = interDir[d];1488
+ MV mvCopy[2];1489
+ for (uint32_t i = 0; i < numDir; i++)1490
+ {1491
+ (analysis->interData)->mvpIdx[i][count + pu] = mvpIdx[i][d];1492
+ (analysis->interData)->refIdx[i][count + pu] = refIdx[i][d];1493
+ mvCopy[i].x = mv[i][d].x * (int16_t)m_param->scaleFactor;1494
+ mvCopy[i].y = mv[i][d].y * (int16_t)m_param->scaleFactor;1495
+ memcpy(&(analysis->interData)->mv[i][count + pu], &mvCopy[i], sizeof(MV));1496
+ }1497
+ }1498
+ d = restoreD; // Restore d value after copying each of the 4 64x64 CTUs1499
+1500
+ if (isScaledMaxCUSize && (puOrient.isRect || puOrient.isAmp))1501
+ {1502
+ /* Skip PU index when current CTU is a 2Nx2N */1503
+ if (partitionSize == SIZE_2Nx2N)1504
+ pu++;1505
+ /* Adjust d after completion of all 4 CTU copies */1506
+ if (numCTU == 3 && (pu == (numPU - 1)))1507
+ d++;1508
+ }1509
+ }1510
+ }1511
+ count += bytes;1512
+ d += getCUIndex(&cuLoc, &count, bytes, 1);1513
+ }1514
+ }1515
+1516
+ X265_FREE(tempBuf);1517
+1518
+ if (m_param->analysisReuseLevel == 10)1519
+ {1520
+ for (uint32_t i = 0; i < numDir; i++)1521
+ {1522
+ X265_FREE(mvpIdx[i]);1523
+ X265_FREE(refIdx[i]);1524
+ X265_FREE(mv[i]);1525
+ }1526
+ if (bIntraInInter)1527
+ {1528
+ cuLoc.evenRowIndex = 0;1529
+ cuLoc.oddRowIndex = m_param->num4x4Partitions * cuLoc.widthInCU;1530
+ cuLoc.switchCondition = 0;1531
+ uint8_t *tempLumaBuf = X265_MALLOC(uint8_t, analysis->numCUsInFrame * scaledNumPartition);1532
+ X265_FREAD(tempLumaBuf, sizeof(uint8_t), analysis->numCUsInFrame * scaledNumPartition, m_analysisFileIn, intraPic->modes);1533
+ uint32_t cnt = 0;1534
+ for (uint32_t ctu32Idx = 0; ctu32Idx < analysis->numCUsInFrame * scaledNumPartition; ctu32Idx++)1535
+ {1536
+ memset(&(analysis->intraData)->modes[cnt], tempLumaBuf[ctu32Idx], factor);1537
+ cnt += factor;1538
+ ctu32Idx += getCUIndex(&cuLoc, &cnt, factor, 0);1539
+ }1540
+ X265_FREE(tempLumaBuf);1541
+ }1542
+ }1543
+ else1544
+ X265_FREAD((analysis->interData)->ref, sizeof(int32_t), analysis->numCUsInFrame * X265_MAX_PRED_MODE_PER_CTU * numDir, m_analysisFileIn, interPic->ref);1545
+1546
+ consumedBytes += frameRecordSize;1547
+ if (numDir == 1)1548
+ totalConsumedBytes = consumedBytes;1549
+ }1550
+1551
+ /* Restore to the current encode's numPartitions and numCUsInFrame */1552
+ analysis->numPartitions = m_param->num4x4Partitions;1553
+ analysis->numCUsInFrame = cuLoc.heightInCU * cuLoc.widthInCU;1554
+ analysis->numCuInHeight = cuLoc.heightInCU;1555
+#undef X265_FREAD1556
+}1557
+1558
+1559
+int Encoder::validateAnalysisData(x265_analysis_data* analysis, int writeFlag)1560
+{1561
+#define X265_PARAM_VALIDATE(analysisParam, size, bytes, param, errorMsg)\1562
+ if(!writeFlag)\1563
+ {\1564
+ fileOffset = m_analysisFileIn;\1565
+ if ((!m_param->bUseAnalysisFile && analysisParam != (int)*param) || \1566
+ (m_param->bUseAnalysisFile && (fread(&readValue, size, bytes, fileOffset) != bytes || (readValue != (int)*param))))\1567
+ {\1568
+ x265_log(NULL, X265_LOG_ERROR, "Error reading analysis data. Incompatible option : <%s> \n", #errorMsg);\1569
+ m_aborted = true;\1570
+ return -1;\1571
+ }\1572
+ }\1573
+ if(writeFlag)\1574
+ {\1575
+ fileOffset = m_analysisFileOut;\1576
+ if(!m_param->bUseAnalysisFile)\1577
+ analysisParam = *param;\1578
+ else if(fwrite(param, size, bytes, fileOffset) < bytes)\1579
+ {\1580
+ x265_log(NULL, X265_LOG_ERROR, "Error writing analysis data\n"); \1581
+ m_aborted = true;\1582
+ return -1; \1583
+ }\1584
+ }\1585
+ count++;1586
+1587
+#define X265_FREAD(val, size, readSize, fileOffset, src)\1588
+ if (!m_param->bUseAnalysisFile)\1589
+ {\1590
+ memcpy(val, src, (size * readSize));\1591
+ }\1592
+ else if (fread(val, size, readSize, fileOffset) != readSize)\1593
+ {\1594
+ x265_log(NULL, X265_LOG_ERROR, "Error reading analysis data\n");\1595
+ m_aborted = true;\1596
+ return -1;\1597
+ }\1598
+ count++;1599
+1600
+ x265_analysis_validate *saveParam = &analysis->saveParam;1601
+ FILE* fileOffset = NULL;1602
+ int readValue = 0;1603
+ int count = 0;1604
+1605
+ X265_PARAM_VALIDATE(saveParam->intraRefresh, sizeof(int), 1, &m_param->bIntraRefresh, intra-refresh);1606
+ X265_PARAM_VALIDATE(saveParam->maxNumReferences, sizeof(int), 1, &m_param->maxNumReferences, ref);1607
+ X265_PARAM_VALIDATE(saveParam->analysisReuseLevel, sizeof(int), 1, &m_param->analysisReuseLevel, analysis-reuse-level);1608
+ X265_PARAM_VALIDATE(saveParam->keyframeMax, sizeof(int), 1, &m_param->keyframeMax, keyint);1609
+ X265_PARAM_VALIDATE(saveParam->keyframeMin, sizeof(int), 1, &m_param->keyframeMin, min-keyint);1610
+ X265_PARAM_VALIDATE(saveParam->openGOP, sizeof(int), 1, &m_param->bOpenGOP, open-gop);1611
+ X265_PARAM_VALIDATE(saveParam->bframes, sizeof(int), 1, &m_param->bframes, bframes);1612
+ X265_PARAM_VALIDATE(saveParam->bPyramid, sizeof(int), 1, &m_param->bBPyramid, bPyramid);1613
+ X265_PARAM_VALIDATE(saveParam->minCUSize, sizeof(int), 1, &m_param->minCUSize, min - cu - size);1614
+ X265_PARAM_VALIDATE(saveParam->lookaheadDepth, sizeof(int), 1, &m_param->lookaheadDepth, rc - lookahead);1615
+ X265_PARAM_VALIDATE(saveParam->chunkStart, sizeof(int), 1, &m_param->chunkStart, chunk-start);1616
+ X265_PARAM_VALIDATE(saveParam->chunkEnd, sizeof(int), 1, &m_param->chunkEnd, chunk-end);1617
+1618
+ int sourceHeight, sourceWidth;1619
+ if (writeFlag)1620
+ {1621
+ sourceHeight = m_param->sourceHeight - m_conformanceWindow.bottomOffset;1622
+ sourceWidth = m_param->sourceWidth - m_conformanceWindow.rightOffset;1623
+ X265_PARAM_VALIDATE(saveParam->sourceWidth, sizeof(int), 1, &sourceWidth, res-width);1624
+ X265_PARAM_VALIDATE(saveParam->sourceHeight, sizeof(int), 1, &sourceHeight, res-height);1625
+ X265_PARAM_VALIDATE(saveParam->maxCUSize, sizeof(int), 1, &m_param->maxCUSize, ctu);1626
+ }1627
+ else1628
+ {1629
+ fileOffset = m_analysisFileIn;1630
+ bool error = false;1631
+ int curSourceHeight = m_param->sourceHeight - m_conformanceWindow.bottomOffset;1632
+ int curSourceWidth = m_param->sourceWidth - m_conformanceWindow.rightOffset;1633
+1634
+ X265_FREAD(&sourceWidth, sizeof(int), 1, m_analysisFileIn, &(saveParam->sourceWidth));1635
+ X265_FREAD(&sourceHeight, sizeof(int), 1, m_analysisFileIn, &(saveParam->sourceHeight));1636
+ X265_FREAD(&readValue, sizeof(int), 1, m_analysisFileIn, &(saveParam->maxCUSize));1637
+1638
+ bool isScaledRes = (2 * sourceHeight == curSourceHeight) && (2 * sourceWidth == curSourceWidth);1639
+ if (!isScaledRes && (sourceHeight != curSourceHeight || sourceWidth != curSourceWidth 1640
+ || readValue != (int)m_param->maxCUSize || m_param->scaleFactor))1641
+ error = true;1642
+ else if (isScaledRes && !m_param->scaleFactor)1643
+ error = true;1644
+ else if (isScaledRes && (int)m_param->maxCUSize == readValue)1645
+ m_saveCTUSize = 1;1646
+ else if (isScaledRes && (g_log2Size[m_param->maxCUSize] - g_log2Size[readValue]) != 1)1647
+ error = true;1648
+1649
+ if (error)1650
+ {1651
+ x265_log(NULL, X265_LOG_ERROR, "Error reading analysis data. Incompatible option : <input-res / scale-factor / ctu> \n");1652
+ m_aborted = true;1653
+ return -1;1654
+ }1655
+ }1656
+ return (count * sizeof(int));1657
+1658
+#undef X265_FREAD1659
+#undef X265_PARAM_VALIDATE1660
+}1661
+1662
+/* Toggle between two consecutive CTU rows. The save's CTU is copied1663
+twice consecutively in the first and second CTU row of load*/1664
+1665
+int Encoder::getCUIndex(cuLocation* cuLoc, uint32_t* count, int bytes, int flag)1666
+{1667
+ int index = 0;1668
+ cuLoc->switchCondition += bytes;1669
+ int isBoundaryW = (*count % (m_param->num4x4Partitions * cuLoc->widthInCU) == 0);1670
+1671
+ /* Width boundary case :1672
+ Skip to appropriate index when out of boundary cases occur1673
+ Out of boundary may occur when the out of bound pixels along1674
+ the width in low resoultion is greater than half of the maxCUSize */1675
+ if (cuLoc->skipWidth && isBoundaryW)1676
+ {1677
+ if (flag)1678
+ index++;1679
+ else1680
+ {1681
+ /* Number of 4x4 blocks in out of bound region */1682
+ int outOfBound = m_param->maxCUSize / 2;1683
+ uint32_t sum = (uint32_t)pow((outOfBound >> 2), 2);1684
+ index += sum;1685
+ }1686
+ cuLoc->switchCondition += m_param->num4x4Partitions;1687
+ }1688
+1689
+ /* Completed writing 2 CTUs - move to the last remembered index of the next CTU row*/1690
+ if (cuLoc->switchCondition == 2 * m_param->num4x4Partitions)1691
+ {1692
+ if (isBoundaryW)1693
+ cuLoc->evenRowIndex = *count + (m_param->num4x4Partitions * cuLoc->widthInCU); // end of row - skip to the next even row1694
+ else1695
+ cuLoc->evenRowIndex = *count;1696
+ *count = cuLoc->oddRowIndex;1697
+1698
+ /* Height boundary case :1699
+ Skip to appropriate index when out of boundary cases occur1700
+ Out of boundary may occur when the out of bound pixels along1701
+ the height in low resoultion is greater than half of the maxCUSize */1702
+ int isBoundaryH = (*count >= (m_param->num4x4Partitions * cuLoc->heightInCU * cuLoc->widthInCU));1703
+ if (cuLoc->skipHeight && isBoundaryH)1704
+ {1705
+ if (flag)1706
+ index += 2;1707
+ else1708
+ {1709
+ int outOfBound = m_param->maxCUSize / 2;1710
+ uint32_t sum = (uint32_t)(2 * pow((abs(outOfBound) >> 2), 2));1711
+ index += sum;1712
+ }1713
+ *count = cuLoc->evenRowIndex;1714
+ cuLoc->switchCondition = 0;1715
+ }1716
+ }1717
+ /* Completed writing 4 CTUs - move to the last remembered index of1718
+ the previous CTU row to copy the next save CTU's data*/1719
+ else if (cuLoc->switchCondition == 4 * m_param->num4x4Partitions)1720
+ {1721
+ if (isBoundaryW)1722
+ cuLoc->oddRowIndex = *count + (m_param->num4x4Partitions * cuLoc->widthInCU); // end of row - skip to the next odd row1723
+ else1724
+ cuLoc->oddRowIndex = *count;1725
+ *count = cuLoc->evenRowIndex;1726
+ cuLoc->switchCondition = 0;1727
+ }1728
+ return index;1729
+}1730
+1731
+/* save load1732
+ CTU0 CTU1 CTU2 CTU31733
+ 2NxN 2Nx2N 2Nx2N 2Nx2N 2Nx2N1734
+ NX2N 2Nx2N 2Nx2N 2Nx2N 2Nx2N1735
+ 2NxnU 2NxN 2NxN 2Nx2N 2Nx2N1736
+ 2NxnD 2Nx2N 2Nx2N 2NxN 2NxN1737
+ nLx2N Nx2N 2Nx2N Nx2N 2Nx2N1738
+ nRx2N 2Nx2N Nx2N 2Nx2N Nx2N1739
+*/1740
+int Encoder::getPuShape(puOrientation* puOrient, int partSize, int numCTU)1741
+{1742
+ puOrient->isRect = true;1743
+ if (partSize == SIZE_Nx2N)1744
+ puOrient->isVert = true;1745
+ if (partSize >= SIZE_2NxnU) // All AMP modes1746
+ {1747
+ puOrient->isAmp = true;1748
+ puOrient->isRect = false;1749
+ if (partSize == SIZE_2NxnD && numCTU > 1)1750
+ return SIZE_2NxN;1751
+ else if (partSize == SIZE_2NxnU && numCTU < 2)1752
+ return SIZE_2NxN;1753
+ else if (partSize == SIZE_nLx2N)1754
+ {1755
+ puOrient->isVert = true;1756
+ if (!(numCTU % 2))1757
+ return SIZE_Nx2N;1758
+ }1759
+ else if (partSize == SIZE_nRx2N)1760
+ {1761
+ puOrient->isVert = true;1762
+ if (numCTU % 2)1763
+ return SIZE_Nx2N;1764
+ }1765
+ }1766
+ return SIZE_2Nx2N;1767
+}1768
+1769
+void Encoder::readAnalysisFile(x265_analysis_data* analysis, int curPoc, int sliceType)1770
{1771
1772
#define X265_FREAD(val, size, readSize, fileOffset)\1773
if (fread(val, size, readSize, fileOffset) != readSize)\1774
{\1775
x265_log(NULL, X265_LOG_ERROR, "Error reading analysis 2 pass data\n"); \1776
- freeAnalysis2Pass(analysis2Pass, sliceType); \1777
+ x265_alloc_analysis_data(m_param, analysis); \1778
m_aborted = true; \1779
return; \1780
}\1781
1782
uint32_t depthBytes = 0;1783
- uint32_t widthInCU = (m_param->sourceWidth + m_param->maxCUSize - 1) >> m_param->maxLog2CUSize;1784
- uint32_t heightInCU = (m_param->sourceHeight + m_param->maxCUSize - 1) >> m_param->maxLog2CUSize;1785
- uint32_t numCUsInFrame = widthInCU * heightInCU;1786
-1787
int poc; uint32_t frameRecordSize;1788
X265_FREAD(&frameRecordSize, sizeof(uint32_t), 1, m_analysisFileIn);1789
X265_FREAD(&depthBytes, sizeof(uint32_t), 1, m_analysisFileIn);1790
1791
if (poc != curPoc || feof(m_analysisFileIn))1792
{1793
x265_log(NULL, X265_LOG_WARNING, "Error reading analysis 2 pass data: Cannot find POC %d\n", curPoc);1794
- freeAnalysis2Pass(analysis2Pass, sliceType);1795
+ x265_free_analysis_data(m_param, analysis);1796
return;1797
}1798
/* Now arrived at the right frame, read the record */1799
- analysis2Pass->frameRecordSize = frameRecordSize;1800
+ analysis->frameRecordSize = frameRecordSize;1801
uint8_t* tempBuf = NULL, *depthBuf = NULL;1802
sse_t *tempdistBuf = NULL, *distortionBuf = NULL;1803
tempBuf = X265_MALLOC(uint8_t, depthBytes);1804
1805
X265_FREAD(tempdistBuf, sizeof(sse_t), depthBytes, m_analysisFileIn);1806
depthBuf = tempBuf;1807
distortionBuf = tempdistBuf;1808
- analysis2PassFrameData* analysisFrameData = (analysis2PassFrameData*)analysis2Pass->analysisFramedata;1809
+ x265_analysis_data *analysisData = (x265_analysis_data*)analysis;1810
+ x265_analysis_intra_data *intraData = analysisData->intraData;1811
+ x265_analysis_inter_data *interData = analysisData->interData;1812
+ x265_analysis_distortion_data *distortionData = analysisData->distortionData;1813
+1814
size_t count = 0;1815
uint32_t ctuCount = 0;1816
double sum = 0, sqrSum = 0;1817
for (uint32_t d = 0; d < depthBytes; d++)1818
{1819
- int bytes = m_param->num4x4Partitions >> (depthBuf[d] * 2);1820
- memset(&analysisFrameData->depth[count], depthBuf[d], bytes);1821
- analysisFrameData->distortion[count] = distortionBuf[d];1822
- analysisFrameData->ctuDistortion[ctuCount] += analysisFrameData->distortion[count];1823
+ int bytes = analysis->numPartitions >> (depthBuf[d] * 2);1824
+ if (IS_X265_TYPE_I(sliceType))1825
+ memset(&intraData->depth[count], depthBuf[d], bytes);1826
+ else1827
+ memset(&interData->depth[count], depthBuf[d], bytes);1828
+ distortionData->distortion[count] = distortionBuf[d];1829
+ distortionData->ctuDistortion[ctuCount] += distortionData->distortion[count];1830
count += bytes;1831
- if ((count % (unsigned)m_param->num4x4Partitions) == 0)1832
+ if ((count % (unsigned)analysis->numPartitions) == 0)1833
{1834
- analysisFrameData->scaledDistortion[ctuCount] = X265_LOG2(X265_MAX(analysisFrameData->ctuDistortion[ctuCount], 1));1835
- sum += analysisFrameData->scaledDistortion[ctuCount];1836
- sqrSum += analysisFrameData->scaledDistortion[ctuCount] * analysisFrameData->scaledDistortion[ctuCount];1837
+ distortionData->scaledDistortion[ctuCount] = X265_LOG2(X265_MAX(distortionData->ctuDistortion[ctuCount], 1));1838
+ sum += distortionData->scaledDistortion[ctuCount];1839
+ sqrSum += distortionData->scaledDistortion[ctuCount] * distortionData->scaledDistortion[ctuCount];1840
ctuCount++;1841
}1842
}1843
- double avg = sum / numCUsInFrame;1844
- analysisFrameData->sdDistortion = pow(((sqrSum / numCUsInFrame) - (avg * avg)), 0.5);1845
- analysisFrameData->averageDistortion = avg;1846
- analysisFrameData->highDistortionCtuCount = analysisFrameData->lowDistortionCtuCount = 0;1847
- for (uint32_t i = 0; i < numCUsInFrame; ++i)1848
- {1849
- analysisFrameData->threshold[i] = analysisFrameData->scaledDistortion[i] / analysisFrameData->averageDistortion;1850
- analysisFrameData->offset[i] = (analysisFrameData->averageDistortion - analysisFrameData->scaledDistortion[i]) / analysisFrameData->sdDistortion;1851
- if (analysisFrameData->threshold[i] < 0.9 && analysisFrameData->offset[i] >= 1)1852
- analysisFrameData->lowDistortionCtuCount++;1853
- else if (analysisFrameData->threshold[i] > 1.1 && analysisFrameData->offset[i] <= -1)1854
- analysisFrameData->highDistortionCtuCount++;1855
+ double avg = sum / analysis->numCUsInFrame;1856
+ distortionData->sdDistortion = pow(((sqrSum / analysis->numCUsInFrame) - (avg * avg)), 0.5);1857
+ distortionData->averageDistortion = avg;1858
+ distortionData->highDistortionCtuCount = distortionData->lowDistortionCtuCount = 0;1859
+ for (uint32_t i = 0; i < analysis->numCUsInFrame; ++i)1860
+ {1861
+ distortionData->threshold[i] = distortionData->scaledDistortion[i] / distortionData->averageDistortion;1862
+ distortionData->offset[i] = (distortionData->averageDistortion - distortionData->scaledDistortion[i]) / distortionData->sdDistortion;1863
+ if (distortionData->threshold[i] < 0.9 && distortionData->offset[i] >= 1)1864
+ distortionData->lowDistortionCtuCount++;1865
+ else if (distortionData->threshold[i] > 1.1 && distortionData->offset[i] <= -1)1866
+ distortionData->highDistortionCtuCount++;1867
}1868
if (!IS_X265_TYPE_I(sliceType))1869
{1870
MV *tempMVBuf[2], *MVBuf[2];1871
- int32_t *tempRefBuf[2], *refBuf[2];1872
- int *tempMvpBuf[2], *mvpBuf[2];1873
+ int32_t *tempRefBuf, *refBuf;1874
+ uint8_t *tempMvpBuf[2], *mvpBuf[2];1875
uint8_t* tempModeBuf = NULL, *modeBuf = NULL;1876
-1877
int numDir = sliceType == X265_TYPE_P ? 1 : 2;1878
+ tempRefBuf = X265_MALLOC(int32_t, numDir * depthBytes);1879
+1880
for (int i = 0; i < numDir; i++)1881
{1882
tempMVBuf[i] = X265_MALLOC(MV, depthBytes);1883
X265_FREAD(tempMVBuf[i], sizeof(MV), depthBytes, m_analysisFileIn);1884
MVBuf[i] = tempMVBuf[i];1885
- tempMvpBuf[i] = X265_MALLOC(int, depthBytes);1886
- X265_FREAD(tempMvpBuf[i], sizeof(int), depthBytes, m_analysisFileIn);1887
+ tempMvpBuf[i] = X265_MALLOC(uint8_t, depthBytes);1888
+ X265_FREAD(tempMvpBuf[i], sizeof(uint8_t), depthBytes, m_analysisFileIn);1889
mvpBuf[i] = tempMvpBuf[i];1890
- tempRefBuf[i] = X265_MALLOC(int32_t, depthBytes);1891
- X265_FREAD(tempRefBuf[i], sizeof(int32_t), depthBytes, m_analysisFileIn);1892
- refBuf[i] = tempRefBuf[i];1893
+ X265_FREAD(&tempRefBuf[i*depthBytes], sizeof(int32_t), depthBytes, m_analysisFileIn);1894
}1895
+ refBuf = tempRefBuf;1896
tempModeBuf = X265_MALLOC(uint8_t, depthBytes);1897
X265_FREAD(tempModeBuf, sizeof(uint8_t), depthBytes, m_analysisFileIn);1898
modeBuf = tempModeBuf;1899
-1900
+ 1901
count = 0;1902
+1903
for (uint32_t d = 0; d < depthBytes; d++)1904
{1905
- size_t bytes = m_param->num4x4Partitions >> (depthBuf[d] * 2);1906
+ size_t bytes = analysis->numPartitions >> (depthBuf[d] * 2);1907
for (int i = 0; i < numDir; i++)1908
{1909
+ int32_t* ref = &(analysis->interData)->ref[i * analysis->numPartitions * analysis->numCUsInFrame];1910
for (size_t j = count, k = 0; k < bytes; j++, k++)1911
{1912
- memcpy(&((analysis2PassFrameData*)analysis2Pass->analysisFramedata)->m_mv[i][j], MVBuf[i] + d, sizeof(MV));1913
- memcpy(&((analysis2PassFrameData*)analysis2Pass->analysisFramedata)->mvpIdx[i][j], mvpBuf[i] + d, sizeof(int));1914
- memcpy(&((analysis2PassFrameData*)analysis2Pass->analysisFramedata)->ref[i][j], refBuf[i] + d, sizeof(int32_t));1915
+ memcpy(&(analysis->interData)->mv[i][j], MVBuf[i] + d, sizeof(MV));1916
+ memcpy(&(analysis->interData)->mvpIdx[i][j], mvpBuf[i] + d, sizeof(uint8_t));1917
+ memcpy(&ref[j], refBuf + (i * depthBytes) + d, sizeof(int32_t));1918
}1919
}1920
- memset(&((analysis2PassFrameData *)analysis2Pass->analysisFramedata)->modes[count], modeBuf[d], bytes);1921
+ memset(&(analysis->interData)->modes[count], modeBuf[d], bytes);1922
count += bytes;1923
}1924
1925
1926
{1927
X265_FREE(tempMVBuf[i]);1928
X265_FREE(tempMvpBuf[i]);1929
- X265_FREE(tempRefBuf[i]);1930
}1931
+ X265_FREE(tempRefBuf);1932
X265_FREE(tempModeBuf);1933
}1934
X265_FREE(tempBuf);1935
1936
if (fwrite(val, size, writeSize, fileOffset) < writeSize)\1937
{\1938
x265_log(NULL, X265_LOG_ERROR, "Error writing analysis data\n");\1939
- freeAnalysis(analysis);\1940
+ x265_free_analysis_data(m_param, analysis);\1941
m_aborted = true;\1942
return;\1943
}\1944
1945
uint32_t numDir, numPlanes;1946
bool bIntraInInter = false;1947
1948
+ if (!analysis->poc)1949
+ {1950
+ if (validateAnalysisData(analysis, 1) == -1)1951
+ {1952
+ m_aborted = true;1953
+ return;1954
+ }1955
+ }1956
+1957
/* calculate frameRecordSize */1958
analysis->frameRecordSize = sizeof(analysis->frameRecordSize) + sizeof(depthBytes) + sizeof(analysis->poc) + sizeof(analysis->sliceType) +1959
sizeof(analysis->numCUsInFrame) + sizeof(analysis->numPartitions) + sizeof(analysis->bScenecut) + sizeof(analysis->satdCost);1960
1961
uint8_t partSize = 0;1962
1963
CUData* ctu = curEncData.getPicCTU(cuAddr);1964
- analysis_intra_data* intraDataCTU = (analysis_intra_data*)analysis->intraData;1965
+ x265_analysis_intra_data* intraDataCTU = analysis->intraData;1966
1967
for (uint32_t absPartIdx = 0; absPartIdx < ctu->m_numPartitions; depthBytes++)1968
{1969
1970
uint8_t partSize = 0;1971
1972
CUData* ctu = curEncData.getPicCTU(cuAddr);1973
- analysis_inter_data* interDataCTU = (analysis_inter_data*)analysis->interData;1974
- analysis_intra_data* intraDataCTU = (analysis_intra_data*)analysis->intraData;1975
+ x265_analysis_inter_data* interDataCTU = analysis->interData;1976
+ x265_analysis_intra_data* intraDataCTU = analysis->intraData;1977
1978
for (uint32_t absPartIdx = 0; absPartIdx < ctu->m_numPartitions; depthBytes++)1979
{1980
1981
{1982
interDataCTU->mvpIdx[dir][depthBytes] = ctu->m_mvpIdx[dir][puabsPartIdx];1983
interDataCTU->refIdx[dir][depthBytes] = ctu->m_refIdx[dir][puabsPartIdx];1984
- interDataCTU->mv[dir][depthBytes] = ctu->m_mv[dir][puabsPartIdx];1985
+ interDataCTU->mv[dir][depthBytes].word = ctu->m_mv[dir][puabsPartIdx].word;1986
}1987
}1988
}1989
1990
1991
if (analysis->sliceType == X265_TYPE_IDR || analysis->sliceType == X265_TYPE_I)1992
{1993
- X265_FWRITE(((analysis_intra_data*)analysis->intraData)->depth, sizeof(uint8_t), depthBytes, m_analysisFileOut);1994
- X265_FWRITE(((analysis_intra_data*)analysis->intraData)->chromaModes, sizeof(uint8_t), depthBytes, m_analysisFileOut);1995
- X265_FWRITE(((analysis_intra_data*)analysis->intraData)->partSizes, sizeof(char), depthBytes, m_analysisFileOut);1996
- X265_FWRITE(((analysis_intra_data*)analysis->intraData)->modes, sizeof(uint8_t), analysis->numCUsInFrame * analysis->numPartitions, m_analysisFileOut);1997
+ X265_FWRITE((analysis->intraData)->depth, sizeof(uint8_t), depthBytes, m_analysisFileOut);1998
+ X265_FWRITE((analysis->intraData)->chromaModes, sizeof(uint8_t), depthBytes, m_analysisFileOut);1999
+ X265_FWRITE((analysis->intraData)->partSizes, sizeof(char), depthBytes, m_analysisFileOut);2000
+ X265_FWRITE((analysis->intraData)->modes, sizeof(uint8_t), analysis->numCUsInFrame * analysis->numPartitions, m_analysisFileOut);2001
}2002
else2003
{2004
- X265_FWRITE(((analysis_inter_data*)analysis->interData)->depth, sizeof(uint8_t), depthBytes, m_analysisFileOut);2005
- X265_FWRITE(((analysis_inter_data*)analysis->interData)->modes, sizeof(uint8_t), depthBytes, m_analysisFileOut);2006
+ X265_FWRITE((analysis->interData)->depth, sizeof(uint8_t), depthBytes, m_analysisFileOut);2007
+ X265_FWRITE((analysis->interData)->modes, sizeof(uint8_t), depthBytes, m_analysisFileOut);2008
if (m_param->analysisReuseLevel > 4)2009
{2010
- X265_FWRITE(((analysis_inter_data*)analysis->interData)->partSize, sizeof(uint8_t), depthBytes, m_analysisFileOut);2011
- X265_FWRITE(((analysis_inter_data*)analysis->interData)->mergeFlag, sizeof(uint8_t), depthBytes, m_analysisFileOut);2012
+ X265_FWRITE((analysis->interData)->partSize, sizeof(uint8_t), depthBytes, m_analysisFileOut);2013
+ X265_FWRITE((analysis->interData)->mergeFlag, sizeof(uint8_t), depthBytes, m_analysisFileOut);2014
if (m_param->analysisReuseLevel == 10)2015
{2016
- X265_FWRITE(((analysis_inter_data*)analysis->interData)->interDir, sizeof(uint8_t), depthBytes, m_analysisFileOut);2017
- if (bIntraInInter) X265_FWRITE(((analysis_intra_data*)analysis->intraData)->chromaModes, sizeof(uint8_t), depthBytes, m_analysisFileOut);2018
+ X265_FWRITE((analysis->interData)->interDir, sizeof(uint8_t), depthBytes, m_analysisFileOut);2019
+ if (bIntraInInter) X265_FWRITE((analysis->intraData)->chromaModes, sizeof(uint8_t), depthBytes, m_analysisFileOut);2020
for (uint32_t dir = 0; dir < numDir; dir++)2021
{2022
- X265_FWRITE(((analysis_inter_data*)analysis->interData)->mvpIdx[dir], sizeof(uint8_t), depthBytes, m_analysisFileOut);2023
- X265_FWRITE(((analysis_inter_data*)analysis->interData)->refIdx[dir], sizeof(int8_t), depthBytes, m_analysisFileOut);2024
- X265_FWRITE(((analysis_inter_data*)analysis->interData)->mv[dir], sizeof(MV), depthBytes, m_analysisFileOut);2025
+ X265_FWRITE((analysis->interData)->mvpIdx[dir], sizeof(uint8_t), depthBytes, m_analysisFileOut);2026
+ X265_FWRITE((analysis->interData)->refIdx[dir], sizeof(int8_t), depthBytes, m_analysisFileOut);2027
+ X265_FWRITE((analysis->interData)->mv[dir], sizeof(MV), depthBytes, m_analysisFileOut);2028
}2029
if (bIntraInInter)2030
- X265_FWRITE(((analysis_intra_data*)analysis->intraData)->modes, sizeof(uint8_t), analysis->numCUsInFrame * analysis->numPartitions, m_analysisFileOut);2031
+ X265_FWRITE((analysis->intraData)->modes, sizeof(uint8_t), analysis->numCUsInFrame * analysis->numPartitions, m_analysisFileOut);2032
}2033
}2034
if (m_param->analysisReuseLevel != 10)2035
- X265_FWRITE(((analysis_inter_data*)analysis->interData)->ref, sizeof(int32_t), analysis->numCUsInFrame * X265_MAX_PRED_MODE_PER_CTU * numDir, m_analysisFileOut);2036
+ X265_FWRITE((analysis->interData)->ref, sizeof(int32_t), analysis->numCUsInFrame * X265_MAX_PRED_MODE_PER_CTU * numDir, m_analysisFileOut);2037
2038
}2039
#undef X265_FWRITE2040
}2041
2042
-void Encoder::writeAnalysis2PassFile(x265_analysis_2Pass* analysis2Pass, FrameData &curEncData, int slicetype)2043
+void Encoder::writeAnalysisFileRefine(x265_analysis_data* analysis, FrameData &curEncData)2044
{2045
#define X265_FWRITE(val, size, writeSize, fileOffset)\2046
if (fwrite(val, size, writeSize, fileOffset) < writeSize)\2047
{\2048
x265_log(NULL, X265_LOG_ERROR, "Error writing analysis 2 pass data\n"); \2049
- freeAnalysis2Pass(analysis2Pass, slicetype); \2050
+ x265_free_analysis_data(m_param, analysis); \2051
m_aborted = true; \2052
return; \2053
}\2054
2055
uint32_t depthBytes = 0;2056
- uint32_t widthInCU = (m_param->sourceWidth + m_param->maxCUSize - 1) >> m_param->maxLog2CUSize;2057
- uint32_t heightInCU = (m_param->sourceHeight + m_param->maxCUSize - 1) >> m_param->maxLog2CUSize;2058
- uint32_t numCUsInFrame = widthInCU * heightInCU;2059
- analysis2PassFrameData* analysisFrameData = (analysis2PassFrameData*)analysis2Pass->analysisFramedata;2060
+ x265_analysis_data *analysisData = (x265_analysis_data*)analysis;2061
+ x265_analysis_intra_data *intraData = analysisData->intraData;2062
+ x265_analysis_inter_data *interData = analysisData->interData;2063
+ x265_analysis_distortion_data *distortionData = analysisData->distortionData;2064
2065
- for (uint32_t cuAddr = 0; cuAddr < numCUsInFrame; cuAddr++)2066
+ for (uint32_t cuAddr = 0; cuAddr < analysis->numCUsInFrame; cuAddr++)2067
{2068
uint8_t depth = 0;2069
2070
2071
for (uint32_t absPartIdx = 0; absPartIdx < ctu->m_numPartitions; depthBytes++)2072
{2073
depth = ctu->m_cuDepth[absPartIdx];2074
- analysisFrameData->depth[depthBytes] = depth;2075
- analysisFrameData->distortion[depthBytes] = ctu->m_distortion[absPartIdx];2076
+ if (curEncData.m_slice->m_sliceType == I_SLICE)2077
+ intraData->depth[depthBytes] = depth;2078
+ else2079
+ interData->depth[depthBytes] = depth;2080
+ distortionData->distortion[depthBytes] = ctu->m_distortion[absPartIdx];2081
absPartIdx += ctu->m_numPartitions >> (depth * 2);2082
}2083
}2084
2085
if (curEncData.m_slice->m_sliceType != I_SLICE)2086
{2087
+ int32_t* ref[2];2088
+ ref[0] = (analysis->interData)->ref;2089
+ ref[1] = &(analysis->interData)->ref[analysis->numPartitions * analysis->numCUsInFrame];2090
depthBytes = 0;2091
- for (uint32_t cuAddr = 0; cuAddr < numCUsInFrame; cuAddr++)2092
+ for (uint32_t cuAddr = 0; cuAddr < analysis->numCUsInFrame; cuAddr++)2093
{2094
uint8_t depth = 0;2095
uint8_t predMode = 0;2096
2097
CUData* ctu = curEncData.getPicCTU(cuAddr);2098
-2099
for (uint32_t absPartIdx = 0; absPartIdx < ctu->m_numPartitions; depthBytes++)2100
{2101
depth = ctu->m_cuDepth[absPartIdx];2102
- analysisFrameData->m_mv[0][depthBytes] = ctu->m_mv[0][absPartIdx];2103
- analysisFrameData->mvpIdx[0][depthBytes] = ctu->m_mvpIdx[0][absPartIdx];2104
- analysisFrameData->ref[0][depthBytes] = ctu->m_refIdx[0][absPartIdx];2105
+ interData->mv[0][depthBytes].word = ctu->m_mv[0][absPartIdx].word;2106
+ interData->mvpIdx[0][depthBytes] = ctu->m_mvpIdx[0][absPartIdx];2107
+ ref[0][depthBytes] = ctu->m_refIdx[0][absPartIdx];2108
predMode = ctu->m_predMode[absPartIdx];2109
if (ctu->m_refIdx[1][absPartIdx] != -1)2110
{2111
- analysisFrameData->m_mv[1][depthBytes] = ctu->m_mv[1][absPartIdx];2112
- analysisFrameData->mvpIdx[1][depthBytes] = ctu->m_mvpIdx[1][absPartIdx];2113
- analysisFrameData->ref[1][depthBytes] = ctu->m_refIdx[1][absPartIdx];2114
+ interData->mv[1][depthBytes].word = ctu->m_mv[1][absPartIdx].word;2115
+ interData->mvpIdx[1][depthBytes] = ctu->m_mvpIdx[1][absPartIdx];2116
+ ref[1][depthBytes] = ctu->m_refIdx[1][absPartIdx];2117
predMode = 4; // used as indiacator if the block is coded as bidir2118
}2119
- analysisFrameData->modes[depthBytes] = predMode;2120
+ interData->modes[depthBytes] = predMode;2121
2122
absPartIdx += ctu->m_numPartitions >> (depth * 2);2123
}2124
2125
}2126
2127
/* calculate frameRecordSize */2128
- analysis2Pass->frameRecordSize = sizeof(analysis2Pass->frameRecordSize) + sizeof(depthBytes) + sizeof(analysis2Pass->poc);2129
-2130
- analysis2Pass->frameRecordSize += depthBytes * sizeof(uint8_t);2131
- analysis2Pass->frameRecordSize += depthBytes * sizeof(sse_t);2132
+ analysis->frameRecordSize = sizeof(analysis->frameRecordSize) + sizeof(depthBytes) + sizeof(analysis->poc);2133
+ analysis->frameRecordSize += depthBytes * sizeof(uint8_t);2134
+ analysis->frameRecordSize += depthBytes * sizeof(sse_t);2135
if (curEncData.m_slice->m_sliceType != I_SLICE)2136
{2137
int numDir = (curEncData.m_slice->m_sliceType == P_SLICE) ? 1 : 2;2138
- analysis2Pass->frameRecordSize += depthBytes * sizeof(MV) * numDir;2139
- analysis2Pass->frameRecordSize += depthBytes * sizeof(int32_t) * numDir;2140
- analysis2Pass->frameRecordSize += depthBytes * sizeof(int) * numDir;2141
- analysis2Pass->frameRecordSize += depthBytes * sizeof(uint8_t);2142
+ analysis->frameRecordSize += depthBytes * sizeof(MV) * numDir;2143
+ analysis->frameRecordSize += depthBytes * sizeof(int32_t) * numDir;2144
+ analysis->frameRecordSize += depthBytes * sizeof(uint8_t) * numDir;2145
+ analysis->frameRecordSize += depthBytes * sizeof(uint8_t);2146
}2147
- X265_FWRITE(&analysis2Pass->frameRecordSize, sizeof(uint32_t), 1, m_analysisFileOut);2148
+ X265_FWRITE(&analysis->frameRecordSize, sizeof(uint32_t), 1, m_analysisFileOut);2149
X265_FWRITE(&depthBytes, sizeof(uint32_t), 1, m_analysisFileOut);2150
- X265_FWRITE(&analysis2Pass->poc, sizeof(uint32_t), 1, m_analysisFileOut);2151
-2152
- X265_FWRITE(analysisFrameData->depth, sizeof(uint8_t), depthBytes, m_analysisFileOut);2153
- X265_FWRITE(analysisFrameData->distortion, sizeof(sse_t), depthBytes, m_analysisFileOut);2154
+ X265_FWRITE(&analysis->poc, sizeof(uint32_t), 1, m_analysisFileOut);2155
+ if (curEncData.m_slice->m_sliceType == I_SLICE)2156
+ {2157
+ X265_FWRITE((analysis->intraData)->depth, sizeof(uint8_t), depthBytes, m_analysisFileOut);2158
+ }2159
+ else2160
+ {2161
+ X265_FWRITE((analysis->interData)->depth, sizeof(uint8_t), depthBytes, m_analysisFileOut);2162
+ }2163
+ X265_FWRITE(distortionData->distortion, sizeof(sse_t), depthBytes, m_analysisFileOut);2164
if (curEncData.m_slice->m_sliceType != I_SLICE)2165
{2166
int numDir = curEncData.m_slice->m_sliceType == P_SLICE ? 1 : 2;2167
for (int i = 0; i < numDir; i++)2168
{2169
- X265_FWRITE(analysisFrameData->m_mv[i], sizeof(MV), depthBytes, m_analysisFileOut);2170
- X265_FWRITE(analysisFrameData->mvpIdx[i], sizeof(int), depthBytes, m_analysisFileOut);2171
- X265_FWRITE(analysisFrameData->ref[i], sizeof(int32_t), depthBytes, m_analysisFileOut);2172
+ int32_t* ref = &(analysis->interData)->ref[i * analysis->numPartitions * analysis->numCUsInFrame];2173
+ X265_FWRITE(interData->mv[i], sizeof(MV), depthBytes, m_analysisFileOut);2174
+ X265_FWRITE(interData->mvpIdx[i], sizeof(uint8_t), depthBytes, m_analysisFileOut);2175
+ X265_FWRITE(ref, sizeof(int32_t), depthBytes, m_analysisFileOut);2176
}2177
- X265_FWRITE(analysisFrameData->modes, sizeof(uint8_t), depthBytes, m_analysisFileOut);2178
+ X265_FWRITE((analysis->interData)->modes, sizeof(uint8_t), depthBytes, m_analysisFileOut);2179
}2180
#undef X265_FWRITE2181
}2182
2183
TOOLCMP(oldParam->rc.rfConstant, newParam->rc.rfConstant, "crf=%f to %f\n");2184
}2185
2186
+void Encoder::readUserSeiFile(x265_sei_payload& seiMsg, int curPoc)2187
+{2188
+ char line[1024];2189
+ while (fgets(line, sizeof(line), m_naluFile))2190
+ {2191
+ int poc = atoi(strtok(line, " "));2192
+ char *prefix = strtok(NULL, " ");2193
+ int nalType = atoi(strtok(NULL, "/"));2194
+ int payloadType = atoi(strtok(NULL, " "));2195
+ char *base64Encode = strtok(NULL, "\n");2196
+ int base64EncodeLength = (int)strlen(base64Encode);2197
+ char *base64Decode = SEI::base64Decode(base64Encode, base64EncodeLength);2198
+ if (nalType == NAL_UNIT_PREFIX_SEI && (!strcmp(prefix, "PREFIX")))2199
+ {2200
+ int currentPOC = curPoc;2201
+ if (currentPOC == poc)2202
+ {2203
+ seiMsg.payloadSize = (base64EncodeLength / 4) * 3;2204
+ seiMsg.payload = (uint8_t*)x265_malloc(sizeof(uint8_t) * seiMsg.payloadSize);2205
+ if (!seiMsg.payload)2206
+ {2207
+ x265_log(m_param, X265_LOG_ERROR, "Unable to allocate memory for SEI payload\n");2208
+ break;2209
+ }2210
+ if (payloadType == 4)2211
+ seiMsg.payloadType = USER_DATA_REGISTERED_ITU_T_T35;2212
+ else if (payloadType == 5)2213
+ seiMsg.payloadType = USER_DATA_UNREGISTERED;2214
+ else2215
+ {2216
+ x265_log(m_param, X265_LOG_WARNING, "Unsupported SEI payload Type for frame %d\n", poc);2217
+ break;2218
+ }2219
+ memcpy(seiMsg.payload, base64Decode, seiMsg.payloadSize);2220
+ break;2221
+ }2222
+ }2223
+ else2224
+ {2225
+ x265_log(m_param, X265_LOG_WARNING, "SEI message for frame %d is not inserted. Will support only PREFIX SEI messages.\n", poc);2226
+ break;2227
+ }2228
+ }2229
+}2230
+2231
bool Encoder::computeSPSRPSIndex()2232
{2233
RPS* rpsInSPS = m_sps.spsrps;2234
x265_2.7.tar.gz/source/encoder/encoder.h -> x265_2.9.tar.gz/source/encoder/encoder.h
Changed
121
1
2
RPSListNode* prior;3
};4
5
+struct cuLocation6
+{7
+ bool skipWidth;8
+ bool skipHeight;9
+ uint32_t heightInCU;10
+ uint32_t widthInCU;11
+ uint32_t oddRowIndex;12
+ uint32_t evenRowIndex;13
+ uint32_t switchCondition;14
+15
+ void init(x265_param* param)16
+ {17
+ skipHeight = false;18
+ skipWidth = false;19
+ heightInCU = (param->sourceHeight + param->maxCUSize - 1) >> param->maxLog2CUSize;20
+ widthInCU = (param->sourceWidth + param->maxCUSize - 1) >> param->maxLog2CUSize;21
+ evenRowIndex = 0;22
+ oddRowIndex = param->num4x4Partitions * widthInCU;23
+ switchCondition = 0; // To switch between odd and even rows24
+ }25
+};26
+27
+struct puOrientation28
+{29
+ bool isVert;30
+ bool isRect;31
+ bool isAmp;32
+33
+ void init()34
+ {35
+ isRect = false;36
+ isAmp = false;37
+ isVert = false;38
+ }39
+};40
+41
+42
class FrameEncoder;43
class DPB;44
class Lookahead;45
46
Frame* m_exportedPic;47
FILE* m_analysisFileIn;48
FILE* m_analysisFileOut;49
+ FILE* m_naluFile;50
x265_param* m_param;51
x265_param* m_latestParam; // Holds latest param during a reconfigure52
RateControl* m_rateControl;53
54
double m_cR;55
56
int m_bToneMap; // Enables tone-mapping57
+ int m_enableNal;58
59
#ifdef ENABLE_HDR10_PLUS60
const hdr10plus_api *m_hdr10plus_api;61
62
63
x265_sei_payload m_prevTonemapPayload;64
65
+ /* Collect frame level feature data */66
+ uint64_t* m_rdCost;67
+ uint64_t* m_variance;68
+ uint32_t* m_trainingCount;69
+ int32_t m_startPoint;70
+ Lock m_dynamicRefineLock;71
+72
+ bool m_saveCTUSize;73
+74
Encoder();75
~Encoder()76
{77
78
79
void updateVbvPlan(RateControl* rc);80
81
- void allocAnalysis(x265_analysis_data* analysis);82
+ void readAnalysisFile(x265_analysis_data* analysis, int poc, int sliceType);83
+84
+ void readAnalysisFile(x265_analysis_data* analysis, int poc, const x265_picture* picIn, int paramBytes);85
86
- void freeAnalysis(x265_analysis_data* analysis);87
+ void readAnalysisFile(x265_analysis_data* analysis, int poc, const x265_picture* picIn, int paramBytes, cuLocation cuLoc);88
89
- void allocAnalysis2Pass(x265_analysis_2Pass* analysis, int sliceType);90
+ int getCUIndex(cuLocation* cuLoc, uint32_t* count, int bytes, int flag);91
92
- void freeAnalysis2Pass(x265_analysis_2Pass* analysis, int sliceType);93
+ int getPuShape(puOrientation* puOrient, int partSize, int numCTU);94
95
- void readAnalysisFile(x265_analysis_data* analysis, int poc, const x265_picture* picIn);96
+ void writeAnalysisFile(x265_analysis_data* analysis, FrameData &curEncData);97
+98
+ void writeAnalysisFileRefine(x265_analysis_data* analysis, FrameData &curEncData);99
100
- void writeAnalysisFile(x265_analysis_data* pic, FrameData &curEncData);101
- void readAnalysis2PassFile(x265_analysis_2Pass* analysis2Pass, int poc, int sliceType);102
- void writeAnalysis2PassFile(x265_analysis_2Pass* analysis2Pass, FrameData &curEncData, int slicetype);103
void finishFrameStats(Frame* pic, FrameEncoder *curEncoder, x265_frame_stats* frameStats, int inPoc);104
105
+ int validateAnalysisData(x265_analysis_data* analysis, int readWriteFlag);106
+107
+ void readUserSeiFile(x265_sei_payload& seiMsg, int poc);108
+109
void calcRefreshInterval(Frame* frameEnc);110
111
void initRefIdx();112
113
void updateRefIdx();114
bool computeSPSRPSIndex();115
116
+ void copyUserSEIMessages(Frame *frame, const x265_picture* pic_in);117
+118
protected:119
120
void initVPS(VPS *vps);121
x265_2.7.tar.gz/source/encoder/entropy.cpp -> x265_2.9.tar.gz/source/encoder/entropy.cpp
Changed
40
1
2
}3
bDenomCoded = true;4
}5
- WRITE_FLAG(wp[0].bPresentFlag, "luma_weight_lX_flag");6
- totalSignalledWeightFlags += wp[0].bPresentFlag;7
+ WRITE_FLAG(!!wp[0].wtPresent, "luma_weight_lX_flag");8
+ totalSignalledWeightFlags += wp[0].wtPresent;9
}10
11
if (bChroma)12
13
for (int ref = 0; ref < slice.m_numRefIdx[list]; ref++)14
{15
wp = slice.m_weightPredTable[list][ref];16
- WRITE_FLAG(wp[1].bPresentFlag, "chroma_weight_lX_flag");17
- totalSignalledWeightFlags += 2 * wp[1].bPresentFlag;18
+ WRITE_FLAG(!!wp[1].wtPresent, "chroma_weight_lX_flag");19
+ totalSignalledWeightFlags += 2 * wp[1].wtPresent;20
}21
}22
23
for (int ref = 0; ref < slice.m_numRefIdx[list]; ref++)24
{25
wp = slice.m_weightPredTable[list][ref];26
- if (wp[0].bPresentFlag)27
+ if (wp[0].wtPresent)28
{29
int deltaWeight = (wp[0].inputWeight - (1 << wp[0].log2WeightDenom));30
WRITE_SVLC(deltaWeight, "delta_luma_weight_lX");31
32
33
if (bChroma)34
{35
- if (wp[1].bPresentFlag)36
+ if (wp[1].wtPresent)37
{38
for (int plane = 1; plane < 3; plane++)39
{40
x265_2.7.tar.gz/source/encoder/frameencoder.cpp -> x265_2.9.tar.gz/source/encoder/frameencoder.cpp
Changed
695
1
2
ok &= m_rce.picTimingSEI && m_rce.hrdTiming;3
}4
5
- if (m_param->noiseReductionIntra || m_param->noiseReductionInter || m_param->rc.vbvBufferSize)6
+ if (m_param->noiseReductionIntra || m_param->noiseReductionInter)7
m_nr = X265_MALLOC(NoiseReduction, 1);8
if (m_nr)9
memset(m_nr, 0, sizeof(NoiseReduction));10
11
return length;12
}13
14
+bool FrameEncoder::writeToneMapInfo(x265_sei_payload *payload)15
+{16
+ bool payloadChange = false;17
+ if (m_top->m_prevTonemapPayload.payload != NULL && payload->payloadSize == m_top->m_prevTonemapPayload.payloadSize)18
+ {19
+ if (memcmp(m_top->m_prevTonemapPayload.payload, payload->payload, payload->payloadSize) != 0)20
+ payloadChange = true;21
+ }22
+ else23
+ {24
+ payloadChange = true;25
+ if (m_top->m_prevTonemapPayload.payload != NULL)26
+ x265_free(m_top->m_prevTonemapPayload.payload);27
+ m_top->m_prevTonemapPayload.payload = (uint8_t*)x265_malloc(sizeof(uint8_t)* payload->payloadSize);28
+ }29
+30
+ if (payloadChange)31
+ {32
+ m_top->m_prevTonemapPayload.payloadType = payload->payloadType;33
+ m_top->m_prevTonemapPayload.payloadSize = payload->payloadSize;34
+ memcpy(m_top->m_prevTonemapPayload.payload, payload->payload, payload->payloadSize);35
+ }36
+37
+ bool isIDR = m_frame->m_lowres.sliceType == X265_TYPE_IDR;38
+ return (payloadChange || isIDR);39
+}40
+41
+void FrameEncoder::writeTrailingSEIMessages()42
+{43
+ Slice* slice = m_frame->m_encData->m_slice;44
+ int planes = (m_param->internalCsp != X265_CSP_I400) ? 3 : 1;45
+ int32_t payloadSize = 0;46
+47
+ if (m_param->decodedPictureHashSEI == 1)48
+ {49
+ m_seiReconPictureDigest.m_method = SEIDecodedPictureHash::MD5;50
+ for (int i = 0; i < planes; i++)51
+ MD5Final(&m_seiReconPictureDigest.m_state[i], m_seiReconPictureDigest.m_digest[i]);52
+ payloadSize = 1 + 16 * planes;53
+ }54
+ else if (m_param->decodedPictureHashSEI == 2)55
+ {56
+ m_seiReconPictureDigest.m_method = SEIDecodedPictureHash::CRC;57
+ for (int i = 0; i < planes; i++)58
+ crcFinish(m_seiReconPictureDigest.m_crc[i], m_seiReconPictureDigest.m_digest[i]);59
+ payloadSize = 1 + 2 * planes;60
+ }61
+ else if (m_param->decodedPictureHashSEI == 3)62
+ {63
+ m_seiReconPictureDigest.m_method = SEIDecodedPictureHash::CHECKSUM;64
+ for (int i = 0; i < planes; i++)65
+ checksumFinish(m_seiReconPictureDigest.m_checksum[i], m_seiReconPictureDigest.m_digest[i]);66
+ payloadSize = 1 + 4 * planes;67
+ }68
+69
+ m_seiReconPictureDigest.setSize(payloadSize);70
+ m_seiReconPictureDigest.writeSEImessages(m_bs, *slice->m_sps, NAL_UNIT_SUFFIX_SEI, m_nalList, false);71
+}72
+73
void FrameEncoder::compressFrame()74
{75
ProfileScopeEvent(frameThread);76
77
* not repeating headers (since AUD is supposed to be the first NAL in the access78
* unit) */79
Slice* slice = m_frame->m_encData->m_slice;80
+81
if (m_param->bEnableAccessUnitDelimiters && (m_frame->m_poc || m_param->bRepeatHeaders))82
{83
m_bs.resetBits();84
85
m_entropyCoder.codeAUD(*slice);86
m_bs.writeByteAlignment();87
m_nalList.serialize(NAL_UNIT_ACCESS_UNIT_DELIMITER, m_bs);88
+ if (m_param->bSingleSeiNal)89
+ m_bs.resetBits();90
}91
if (m_frame->m_lowres.bKeyframe && m_param->bRepeatHeaders)92
{93
94
wa.waitForExit();95
else96
weightAnalyse(*slice, *m_frame, *m_param);97
-98
}99
-100
}101
else102
slice->disableWeights();103
104
for (int ref = 0; ref < slice->m_numRefIdx[l]; ref++)105
{106
WeightParam *w = NULL;107
- if ((bUseWeightP || bUseWeightB) && slice->m_weightPredTable[l][ref][0].bPresentFlag)108
+ if ((bUseWeightP || bUseWeightB) && slice->m_weightPredTable[l][ref][0].wtPresent)109
w = slice->m_weightPredTable[l][ref];110
slice->m_refReconPicList[l][ref] = slice->m_refFrameList[l][ref]->m_reconPic;111
m_mref[l][ref].init(slice->m_refReconPicList[l][ref], w, *m_param);112
113
114
/* Get the QP for this frame from rate control. This call may block until115
* frames ahead of it in encode order have called rateControlEnd() */116
- m_rce.encodeOrder = m_frame->m_encodeOrder;117
- bool payloadChange = false;118
- bool writeSei = true;119
- if (m_param->bDhdr10opt)120
- {121
- for (int i = 0; i < m_frame->m_userSEI.numPayloads; i++)122
- {123
- x265_sei_payload *payload = &m_frame->m_userSEI.payloads[i];124
- if(payload->payloadType == USER_DATA_REGISTERED_ITU_T_T35)125
- {126
- if (m_top->m_prevTonemapPayload.payload != NULL && payload->payloadSize == m_top->m_prevTonemapPayload.payloadSize)127
- {128
- if (memcmp(m_top->m_prevTonemapPayload.payload, payload->payload, payload->payloadSize) != 0)129
- payloadChange = true;130
- }131
- else132
- {133
- payloadChange = true;134
- if (m_top->m_prevTonemapPayload.payload != NULL)135
- x265_free(m_top->m_prevTonemapPayload.payload);136
- m_top->m_prevTonemapPayload.payload = (uint8_t*)x265_malloc(sizeof(uint8_t) * payload->payloadSize);137
- }138
-139
- if (payloadChange)140
- {141
- m_top->m_prevTonemapPayload.payloadType = payload->payloadType;142
- m_top->m_prevTonemapPayload.payloadSize = payload->payloadSize;143
- memcpy(m_top->m_prevTonemapPayload.payload, payload->payload, payload->payloadSize);144
- }145
-146
- bool isIDR = m_frame->m_lowres.sliceType == X265_TYPE_IDR;147
- writeSei = payloadChange || isIDR;148
- }149
- }150
- }151
int qp = m_top->m_rateControl->rateControlStart(m_frame, &m_rce, m_top);152
m_rce.newQp = qp;153
154
155
156
/* reset entropy coders and compute slice id */157
m_entropyCoder.load(m_initSliceContext);158
- 159
for (uint32_t sliceId = 0; sliceId < m_param->maxSlices; sliceId++) 160
for (uint32_t row = m_sliceBaseRow[sliceId]; row < m_sliceBaseRow[sliceId + 1]; row++)161
m_rows[row].init(m_initSliceContext, sliceId); 162
163
m_outStreams[i].resetBits();164
}165
166
+ m_rce.encodeOrder = m_frame->m_encodeOrder;167
int prevBPSEI = m_rce.encodeOrder ? m_top->m_lastBPSEI : 0;168
169
if (m_frame->m_lowres.bKeyframe)170
171
bpSei->m_auCpbRemovalDelayDelta = 1;172
bpSei->m_cpbDelayOffset = 0;173
bpSei->m_dpbDelayOffset = 0;174
-175
// hrdFullness() calculates the initial CPB removal delay and offset176
m_top->m_rateControl->hrdFullness(bpSei);177
-178
- m_bs.resetBits();179
- bpSei->write(m_bs, *slice->m_sps);180
- m_bs.writeByteAlignment();181
-182
- m_nalList.serialize(NAL_UNIT_PREFIX_SEI, m_bs);183
+ bpSei->writeSEImessages(m_bs, *slice->m_sps, NAL_UNIT_PREFIX_SEI, m_nalList, m_param->bSingleSeiNal);184
185
m_top->m_lastBPSEI = m_rce.encodeOrder;186
}187
+188
+ if (m_frame->m_lowres.sliceType == X265_TYPE_IDR && m_param->bEmitIDRRecoverySEI)189
+ {190
+ /* Recovery Point SEI require the SPS to be "activated" */191
+ SEIRecoveryPoint sei;192
+ sei.m_recoveryPocCnt = 0;193
+ sei.m_exactMatchingFlag = true;194
+ sei.m_brokenLinkFlag = false;195
+ sei.writeSEImessages(m_bs, *slice->m_sps, NAL_UNIT_PREFIX_SEI, m_nalList, m_param->bSingleSeiNal);196
+ }197
}198
199
if ((m_param->bEmitHRDSEI || !!m_param->interlaceMode))200
201
else if (m_param->interlaceMode == 1)202
sei->m_picStruct = (poc & 1) ? 2 /* bottom */ : 1 /* top */;203
else204
- sei->m_picStruct = 0;205
- sei->m_sourceScanType = 0;206
+ sei->m_picStruct = m_param->pictureStructure;207
+208
+ sei->m_sourceScanType = m_param->interlaceMode ? 0 : 1;209
+210
sei->m_duplicateFlag = false;211
}212
213
214
sei->m_picDpbOutputDelay = slice->m_sps->numReorderPics + poc - m_rce.encodeOrder;215
}216
217
- m_bs.resetBits();218
- sei->write(m_bs, *slice->m_sps);219
- m_bs.writeByteAlignment();220
- m_nalList.serialize(NAL_UNIT_PREFIX_SEI, m_bs);221
+ sei->writeSEImessages(m_bs, *slice->m_sps, NAL_UNIT_PREFIX_SEI, m_nalList, m_param->bSingleSeiNal);222
+ }223
+224
+ if (m_param->preferredTransferCharacteristics > -1 && slice->isIRAP())225
+ {226
+ SEIAlternativeTC m_seiAlternativeTC;227
+ m_seiAlternativeTC.m_preferredTransferCharacteristics = m_param->preferredTransferCharacteristics;228
+ m_seiAlternativeTC.writeSEImessages(m_bs, *slice->m_sps, NAL_UNIT_PREFIX_SEI, m_nalList, m_param->bSingleSeiNal);229
}230
231
/* Write user SEI */232
233
{234
SEIuserDataUnregistered sei;235
sei.m_userData = payload->payload;236
- m_bs.resetBits();237
sei.setSize(payload->payloadSize);238
- sei.write(m_bs, *slice->m_sps);239
- m_bs.writeByteAlignment();240
- m_nalList.serialize(NAL_UNIT_PREFIX_SEI, m_bs);241
+ sei.writeSEImessages(m_bs, *slice->m_sps, NAL_UNIT_PREFIX_SEI, m_nalList, m_param->bSingleSeiNal);242
}243
else if (payload->payloadType == USER_DATA_REGISTERED_ITU_T_T35)244
{245
+ bool writeSei = m_param->bDhdr10opt ? writeToneMapInfo(payload) : true;246
if (writeSei)247
{248
- SEICreativeIntentMeta sei;249
- sei.m_payload = payload->payload;250
- m_bs.resetBits();251
+ SEIuserDataRegistered sei;252
+ sei.m_userData = payload->payload;253
sei.setSize(payload->payloadSize);254
- sei.write(m_bs, *slice->m_sps);255
- m_bs.writeByteAlignment();256
- m_nalList.serialize(NAL_UNIT_PREFIX_SEI, m_bs);257
+ sei.writeSEImessages(m_bs, *slice->m_sps, NAL_UNIT_PREFIX_SEI, m_nalList, m_param->bSingleSeiNal);258
}259
}260
else261
x265_log(m_param, X265_LOG_ERROR, "Unrecognized SEI type\n");262
}263
+264
+ bool isSei = ((m_frame->m_lowres.bKeyframe && m_param->bRepeatHeaders) || m_param->bEmitHRDSEI ||265
+ !!m_param->interlaceMode || (m_frame->m_lowres.sliceType == X265_TYPE_IDR && m_param->bEmitIDRRecoverySEI) ||266
+ m_frame->m_userSEI.numPayloads);267
+268
+ if (isSei && m_param->bSingleSeiNal)269
+ {270
+ m_bs.writeByteAlignment();271
+ m_nalList.serialize(NAL_UNIT_PREFIX_SEI, m_bs);272
+ }273
/* CQP and CRF (without capped VBV) doesn't use mid-frame statistics to 274
* tune RateControl parameters for other frames.275
* Hence, for these modes, update m_startEndOrder and unlock RC for previous threads waiting in276
277
m_top->m_rateControl->m_startEndOrder.incr(); // faked rateControlEnd calls for negative frames278
}279
280
+ if (m_param->bDynamicRefine)281
+ computeAvgTrainingData();282
+283
/* Analyze CTU rows, most of the hard work is done here. Frame is284
* compressed in a wave-front pattern if WPP is enabled. Row based loop285
* filters runs behind the CTU compression and reconstruction */286
287
m_frameFilter.processRow(i - m_filterRowDelay);288
}289
}290
+#if ENABLE_LIBVMAF291
+ vmafFrameLevelScore();292
+#endif293
294
if (m_param->maxSlices > 1)295
{296
PicYuv *reconPic = m_frame->m_reconPic;297
uint32_t height = reconPic->m_picHeight;298
- uint32_t width = reconPic->m_picWidth;299
- intptr_t stride = reconPic->m_stride;300
- const uint32_t hChromaShift = CHROMA_H_SHIFT(m_param->internalCsp);301
- const uint32_t vChromaShift = CHROMA_V_SHIFT(m_param->internalCsp);302
+ initDecodedPictureHashSEI(0, 0, height);303
+ } 304
305
- if (m_param->decodedPictureHashSEI == 1)306
- {307
-308
- MD5Init(&m_state[0]);309
-310
- updateMD5Plane(m_state[0], reconPic->m_picOrg[0], width, height, stride);311
-312
- if (m_param->internalCsp != X265_CSP_I400)313
- {314
- MD5Init(&m_state[1]);315
- MD5Init(&m_state[2]);316
-317
- width >>= hChromaShift;318
- height >>= vChromaShift;319
- stride = reconPic->m_strideC;320
-321
- updateMD5Plane(m_state[1], reconPic->m_picOrg[1], width, height, stride);322
- updateMD5Plane(m_state[2], reconPic->m_picOrg[2], width, height, stride);323
- }324
- }325
- // TODO: NOT verify code in below mode326
- else if (m_param->decodedPictureHashSEI == 2)327
- {328
- m_crc[0] = 0xffff;329
-330
- updateCRC(reconPic->m_picOrg[0], m_crc[0], height, width, stride);331
-332
- if (m_param->internalCsp != X265_CSP_I400)333
- {334
- width >>= hChromaShift;335
- height >>= vChromaShift;336
- stride = reconPic->m_strideC;337
- m_crc[1] = m_crc[2] = 0xffff;338
-339
- updateCRC(reconPic->m_picOrg[1], m_crc[1], height, width, stride);340
- updateCRC(reconPic->m_picOrg[2], m_crc[2], height, width, stride);341
- }342
- }343
- else if (m_param->decodedPictureHashSEI == 3)344
- {345
- uint32_t cuHeight = m_param->maxCUSize;346
-347
- m_checksum[0] = 0;348
-349
- updateChecksum(reconPic->m_picOrg[0], m_checksum[0], height, width, stride, 0, cuHeight);350
-351
- if (m_param->internalCsp != X265_CSP_I400)352
- {353
- width >>= hChromaShift;354
- height >>= vChromaShift;355
- stride = reconPic->m_strideC;356
- cuHeight >>= vChromaShift;357
-358
- m_checksum[1] = m_checksum[2] = 0;359
-360
- updateChecksum(reconPic->m_picOrg[1], m_checksum[1], height, width, stride, 0, cuHeight);361
- updateChecksum(reconPic->m_picOrg[2], m_checksum[2], height, width, stride, 0, cuHeight);362
- }363
- }364
- } // end of (m_param->maxSlices > 1)365
+ if (m_param->bDynamicRefine && m_top->m_startPoint <= m_frame->m_encodeOrder) //Avoid collecting data that will not be used by future frames.366
+ collectDynDataFrame();367
368
if (m_param->rc.bStatWrite)369
{370
371
m_bs.resetBits();372
373
const uint32_t sliceAddr = nextSliceRow * m_numCols;374
- //CUData* ctu = m_frame->m_encData->getPicCTU(sliceAddr);375
- //const int sliceQp = ctu->m_qp[0];376
if (m_param->bOptRefListLengthPPS)377
{378
ScopedLock refIdxLock(m_top->m_sliceRefIdxLock);379
380
m_nalList.serialize(slice->m_nalUnitType, m_bs);381
}382
383
-384
if (m_param->decodedPictureHashSEI)385
- {386
- int planes = (m_frame->m_param->internalCsp != X265_CSP_I400) ? 3 : 1;387
- int32_t payloadSize = 0;388
- if (m_param->decodedPictureHashSEI == 1)389
- {390
- m_seiReconPictureDigest.m_method = SEIDecodedPictureHash::MD5;391
- for (int i = 0; i < planes; i++)392
- MD5Final(&m_state[i], m_seiReconPictureDigest.m_digest[i]);393
- payloadSize = 1 + 16 * planes;394
- }395
- else if (m_param->decodedPictureHashSEI == 2)396
- {397
- m_seiReconPictureDigest.m_method = SEIDecodedPictureHash::CRC;398
- for (int i = 0; i < planes; i++)399
- crcFinish(m_crc[i], m_seiReconPictureDigest.m_digest[i]);400
- payloadSize = 1 + 2 * planes;401
- }402
- else if (m_param->decodedPictureHashSEI == 3)403
- {404
- m_seiReconPictureDigest.m_method = SEIDecodedPictureHash::CHECKSUM;405
- for (int i = 0; i < planes; i++)406
- checksumFinish(m_checksum[i], m_seiReconPictureDigest.m_digest[i]);407
- payloadSize = 1 + 4 * planes;408
- }409
- m_bs.resetBits();410
- m_seiReconPictureDigest.setSize(payloadSize);411
- m_seiReconPictureDigest.write(m_bs, *slice->m_sps);412
- m_bs.writeByteAlignment();413
- m_nalList.serialize(NAL_UNIT_SUFFIX_SEI, m_bs);414
- }415
+ writeTrailingSEIMessages();416
417
uint64_t bytes = 0;418
for (uint32_t i = 0; i < m_nalList.m_numNal; i++)419
420
m_cuStats.accumulate(m_tld[i].analysis.m_stats[m_jpId], *m_param);421
#endif422
423
- m_endFrameTime = x265_mdate();424
+ m_endFrameTime = x265_mdate(); 425
+}426
+427
+void FrameEncoder::initDecodedPictureHashSEI(int row, int cuAddr, int height)428
+{429
+ PicYuv *reconPic = m_frame->m_reconPic;430
+ uint32_t width = reconPic->m_picWidth; 431
+ intptr_t stride = reconPic->m_stride;432
+ uint32_t maxCUHeight = m_param->maxCUSize;433
+434
+ const uint32_t hChromaShift = CHROMA_H_SHIFT(m_param->internalCsp);435
+ const uint32_t vChromaShift = CHROMA_V_SHIFT(m_param->internalCsp);436
+437
+ if (m_param->decodedPictureHashSEI == 1)438
+ {439
+ if (!row)440
+ MD5Init(&m_seiReconPictureDigest.m_state[0]);441
+442
+ updateMD5Plane(m_seiReconPictureDigest.m_state[0], reconPic->getLumaAddr(cuAddr), width, height, stride);443
+ if (m_param->internalCsp != X265_CSP_I400)444
+ {445
+ if (!row)446
+ {447
+ MD5Init(&m_seiReconPictureDigest.m_state[1]);448
+ MD5Init(&m_seiReconPictureDigest.m_state[2]);449
+ }450
+451
+ width >>= hChromaShift;452
+ height >>= vChromaShift;453
+ stride = reconPic->m_strideC;454
+455
+ updateMD5Plane(m_seiReconPictureDigest.m_state[1], reconPic->getCbAddr(cuAddr), width, height, stride);456
+ updateMD5Plane(m_seiReconPictureDigest.m_state[2], reconPic->getCrAddr(cuAddr), width, height, stride);457
+ }458
+ }459
+ else if (m_param->decodedPictureHashSEI == 2)460
+ {461
+462
+ if (!row)463
+ m_seiReconPictureDigest.m_crc[0] = 0xffff;464
+465
+ updateCRC(reconPic->getLumaAddr(cuAddr), m_seiReconPictureDigest.m_crc[0], height, width, stride);466
+ if (m_param->internalCsp != X265_CSP_I400)467
+ {468
+ width >>= hChromaShift;469
+ height >>= vChromaShift;470
+ stride = reconPic->m_strideC;471
+ m_seiReconPictureDigest.m_crc[1] = m_seiReconPictureDigest.m_crc[2] = 0xffff;472
+473
+ updateCRC(reconPic->getCbAddr(cuAddr), m_seiReconPictureDigest.m_crc[1], height, width, stride);474
+ updateCRC(reconPic->getCrAddr(cuAddr), m_seiReconPictureDigest.m_crc[2], height, width, stride);475
+ }476
+ }477
+ else if (m_param->decodedPictureHashSEI == 3)478
+ {479
+ if (!row)480
+ m_seiReconPictureDigest.m_checksum[0] = 0;481
+482
+ updateChecksum(reconPic->m_picOrg[0], m_seiReconPictureDigest.m_checksum[0], height, width, stride, row, maxCUHeight);483
+ if (m_param->internalCsp != X265_CSP_I400)484
+ {485
+ width >>= hChromaShift;486
+ height >>= vChromaShift;487
+ stride = reconPic->m_strideC;488
+ maxCUHeight >>= vChromaShift;489
+490
+ if (!row)491
+ m_seiReconPictureDigest.m_checksum[1] = m_seiReconPictureDigest.m_checksum[2] = 0;492
+493
+ updateChecksum(reconPic->m_picOrg[1], m_seiReconPictureDigest.m_checksum[1], height, width, stride, row, maxCUHeight);494
+ updateChecksum(reconPic->m_picOrg[2], m_seiReconPictureDigest.m_checksum[2], height, width, stride, row, maxCUHeight);495
+ }496
+ }497
}498
499
void FrameEncoder::encodeSlice(uint32_t sliceAddr)500
501
}502
curRow.avgQPComputed = 1;503
}504
- } 505
+ }506
507
// Initialize restrict on MV range in slices508
tld.analysis.m_sliceMinY = -(int16_t)(rowInSlice * m_param->maxCUSize * 4) + 3 * 4;509
510
// Does all the CU analysis, returns best top level mode decision511
Mode& best = tld.analysis.compressCTU(*ctu, *m_frame, m_cuGeoms[m_ctuGeomMap[cuAddr]], rowCoder);512
513
+ /* startPoint > encodeOrder is true when the start point changes for514
+ a new GOP but few frames from the previous GOP is still incomplete.515
+ The data of frames in this interval will not be used by any future frames. */516
+ if (m_param->bDynamicRefine && m_top->m_startPoint <= m_frame->m_encodeOrder)517
+ collectDynDataRow(*ctu, &curRow.rowStats);518
+519
// take a sample of the current active worker count520
ATOMIC_ADD(&m_totalActiveWorkerCount, m_activeWorkerCount);521
ATOMIC_INC(&m_activeWorkerCountSamples);522
523
{524
// NOTE: in VBV mode, we may reencode anytime, so we can't do Deblock stage-Horizon and SAO525
if (!bIsVbv)526
- { 527
+ {528
// Delay one row to avoid intra prediction conflict529
if (m_pool && !bFirstRowInSlice)530
{ 531
532
else if ((uint32_t)m_rce.encodeOrder <= 2 * (m_param->fpsNum / m_param->fpsDenom))533
rowCount = X265_MIN((maxRows + 1) / 2, maxRows - 1);534
else535
- rowCount = X265_MIN(m_refLagRows / m_param->maxSlices, maxRows - 1);536
+ rowCount = X265_MIN(m_refLagRows / m_param->maxSlices, maxRows - 1);537
538
if (rowInSlice == rowCount)539
{540
m_rowSliceTotalBits[sliceId] = 0;541
if (bIsVbv && !(m_param->rc.bEnableConstVbv && m_param->bEnableWavefront))542
- { 543
+ {544
for (uint32_t i = m_sliceBaseRow[sliceId]; i < rowCount + m_sliceBaseRow[sliceId]; i++)545
m_rowSliceTotalBits[sliceId] += curEncData.m_rowStat[i].encodedBits;546
}547
else548
{549
uint32_t startAddr = m_sliceBaseRow[sliceId] * numCols;550
- uint32_t finishAddr = startAddr + rowCount * numCols;551
+ uint32_t finishAddr = startAddr + rowCount * numCols;552
553
- for (uint32_t cuAddr = startAddr; cuAddr < finishAddr; cuAddr++)554
+ for (uint32_t cuAddr = startAddr; cuAddr < finishAddr; cuAddr++)555
m_rowSliceTotalBits[sliceId] += curEncData.m_cuStat[cuAddr].totalBits;556
- } 557
+ }558
559
if (ATOMIC_INC(&m_sliceCnt) == (int)m_param->maxSlices)560
{561
562
m_completionEvent.trigger();563
}564
565
+void FrameEncoder::collectDynDataRow(CUData& ctu, FrameStats* rowStats)566
+{567
+ for (uint32_t i = 0; i < X265_REFINE_INTER_LEVELS; i++)568
+ {569
+ for (uint32_t depth = 0; depth < m_param->maxCUDepth; depth++)570
+ {571
+ int offset = (depth * X265_REFINE_INTER_LEVELS) + i;572
+ if (ctu.m_collectCUCount[offset])573
+ {574
+ rowStats->rowVarDyn[offset] += ctu.m_collectCUVariance[offset];575
+ rowStats->rowRdDyn[offset] += ctu.m_collectCURd[offset];576
+ rowStats->rowCntDyn[offset] += ctu.m_collectCUCount[offset];577
+ }578
+ }579
+ }580
+}581
+582
+void FrameEncoder::collectDynDataFrame()583
+{584
+ for (uint32_t row = 0; row < m_numRows; row++)585
+ {586
+ for (uint32_t refLevel = 0; refLevel < X265_REFINE_INTER_LEVELS; refLevel++)587
+ {588
+ for (uint32_t depth = 0; depth < m_param->maxCUDepth; depth++)589
+ {590
+ int offset = (depth * X265_REFINE_INTER_LEVELS) + refLevel;591
+ int curFrameIndex = m_frame->m_encodeOrder - m_top->m_startPoint;592
+ int index = (curFrameIndex * X265_REFINE_INTER_LEVELS * m_param->maxCUDepth) + offset;593
+ if (m_rows[row].rowStats.rowCntDyn[offset])594
+ {595
+ m_top->m_variance[index] += m_rows[row].rowStats.rowVarDyn[offset];596
+ m_top->m_rdCost[index] += m_rows[row].rowStats.rowRdDyn[offset];597
+ m_top->m_trainingCount[index] += m_rows[row].rowStats.rowCntDyn[offset];598
+ }599
+ }600
+ }601
+ }602
+}603
+604
+void FrameEncoder::computeAvgTrainingData()605
+{606
+ if (m_frame->m_lowres.bScenecut || m_frame->m_lowres.bKeyframe)607
+ {608
+ m_top->m_startPoint = m_frame->m_encodeOrder;609
+ int size = (m_param->keyframeMax + m_param->lookaheadDepth) * m_param->maxCUDepth * X265_REFINE_INTER_LEVELS;610
+ memset(m_top->m_variance, 0, size * sizeof(uint64_t));611
+ memset(m_top->m_rdCost, 0, size * sizeof(uint64_t));612
+ memset(m_top->m_trainingCount, 0, size * sizeof(uint32_t));613
+ }614
+ if (m_frame->m_encodeOrder - m_top->m_startPoint < 2 * m_param->frameNumThreads)615
+ m_frame->m_classifyFrame = false;616
+ else617
+ m_frame->m_classifyFrame = true;618
+619
+ int size = m_param->maxCUDepth * X265_REFINE_INTER_LEVELS;620
+ memset(m_frame->m_classifyRd, 0, size * sizeof(uint64_t));621
+ memset(m_frame->m_classifyVariance, 0, size * sizeof(uint64_t));622
+ memset(m_frame->m_classifyCount, 0, size * sizeof(uint32_t));623
+ if (m_frame->m_classifyFrame)624
+ {625
+ uint32_t limit = m_frame->m_encodeOrder - m_top->m_startPoint - m_param->frameNumThreads;626
+ for (uint32_t i = 1; i < limit; i++)627
+ {628
+ for (uint32_t j = 0; j < X265_REFINE_INTER_LEVELS; j++)629
+ {630
+ for (uint32_t depth = 0; depth < m_param->maxCUDepth; depth++)631
+ {632
+ int offset = (depth * X265_REFINE_INTER_LEVELS) + j;633
+ int index = (i* X265_REFINE_INTER_LEVELS * m_param->maxCUDepth) + offset;634
+ if (m_top->m_trainingCount[index])635
+ {636
+ m_frame->m_classifyRd[offset] += m_top->m_rdCost[index] / m_top->m_trainingCount[index];637
+ m_frame->m_classifyVariance[offset] += m_top->m_variance[index] / m_top->m_trainingCount[index];638
+ m_frame->m_classifyCount[offset] += m_top->m_trainingCount[index];639
+ }640
+ }641
+ }642
+ }643
+ /* Calculates the average feature values of historic frames that are being considered for the current frame */644
+ int historyCount = m_frame->m_encodeOrder - m_param->frameNumThreads - m_top->m_startPoint - 1;645
+ if (historyCount)646
+ {647
+ for (uint32_t j = 0; j < X265_REFINE_INTER_LEVELS; j++)648
+ {649
+ for (uint32_t depth = 0; depth < m_param->maxCUDepth; depth++)650
+ {651
+ int offset = (depth * X265_REFINE_INTER_LEVELS) + j;652
+ m_frame->m_classifyRd[offset] /= historyCount;653
+ m_frame->m_classifyVariance[offset] /= historyCount;654
+ }655
+ }656
+ }657
+ }658
+}659
+660
/* collect statistics about CU coding decisions, return total QP */661
int FrameEncoder::collectCTUStatistics(const CUData& ctu, FrameStats* log)662
{663
664
m_nr->nrOffsetDenoise[cat][0] = 0;665
}666
}667
+#if ENABLE_LIBVMAF668
+void FrameEncoder::vmafFrameLevelScore()669
+{670
+ PicYuv *fenc = m_frame->m_fencPic;671
+ PicYuv *recon = m_frame->m_reconPic;672
+673
+ x265_vmaf_framedata *vmafframedata = (x265_vmaf_framedata*)x265_malloc(sizeof(x265_vmaf_framedata));674
+ if (!vmafframedata)675
+ {676
+ x265_log(NULL, X265_LOG_ERROR, "vmaf frame data alloc failed\n");677
+ }678
+679
+ vmafframedata->height = fenc->m_picHeight;680
+ vmafframedata->width = fenc->m_picWidth;681
+ vmafframedata->frame_set = 0;682
+ vmafframedata->internalBitDepth = m_param->internalBitDepth;683
+ vmafframedata->reference_frame = fenc;684
+ vmafframedata->distorted_frame = recon;685
+686
+ fenc->m_vmafScore = x265_calculate_vmaf_framelevelscore(vmafframedata);687
+688
+ if (vmafframedata)689
+ x265_free(vmafframedata);690
+}691
+#endif692
693
Frame *FrameEncoder::getEncodedPicture(NALList& output)694
{695
x265_2.7.tar.gz/source/encoder/frameencoder.h -> x265_2.9.tar.gz/source/encoder/frameencoder.h
Changed
42
1
2
/* blocks until worker thread is done, returns access unit */3
Frame *getEncodedPicture(NALList& list);4
5
+ void initDecodedPictureHashSEI(int row, int cuAddr, int height);6
+7
Event m_enable;8
Event m_done;9
Event m_completionEvent;10
11
double m_ssim;12
uint64_t m_accessUnitBits;13
uint32_t m_ssimCnt;14
- MD5Context m_state[3];15
- uint32_t m_crc[3];16
- uint32_t m_checksum[3];17
18
volatile int m_activeWorkerCount; // count of workers currently encoding or filtering CTUs19
volatile int m_totalActiveWorkerCount; // sum of m_activeWorkerCount sampled at end of each CTU20
21
void threadMain();22
int collectCTUStatistics(const CUData& ctu, FrameStats* frameLog);23
void noiseReductionUpdate();24
+ void writeTrailingSEIMessages();25
+ bool writeToneMapInfo(x265_sei_payload *payload);26
27
/* Called by WaveFront::findJob() */28
virtual void processRow(int row, int threadId);29
30
void enqueueRowFilter(int row) { WaveFront::enqueueRow(row * 2 + 1); }31
void enableRowEncoder(int row) { WaveFront::enableRow(row * 2 + 0); }32
void enableRowFilter(int row) { WaveFront::enableRow(row * 2 + 1); }33
+#if ENABLE_LIBVMAF34
+ void vmafFrameLevelScore();35
+#endif36
+ void collectDynDataFrame();37
+ void computeAvgTrainingData();38
+ void collectDynDataRow(CUData& ctu, FrameStats* rowStats); 39
};40
}41
42
x265_2.7.tar.gz/source/encoder/framefilter.cpp -> x265_2.9.tar.gz/source/encoder/framefilter.cpp
Changed
82
1
2
3
if (m_param->maxSlices == 1)4
{5
- if (m_param->decodedPictureHashSEI == 1)6
- {7
- uint32_t height = m_parallelFilter[row].getCUHeight();8
- uint32_t width = reconPic->m_picWidth;9
- intptr_t stride = reconPic->m_stride;10
-11
- if (!row)12
- MD5Init(&m_frameEncoder->m_state[0]);13
-14
- updateMD5Plane(m_frameEncoder->m_state[0], reconPic->getLumaAddr(cuAddr), width, height, stride);15
- if (m_param->internalCsp != X265_CSP_I400)16
- {17
- if (!row)18
- {19
- MD5Init(&m_frameEncoder->m_state[1]);20
- MD5Init(&m_frameEncoder->m_state[2]);21
- }22
-23
- width >>= m_hChromaShift;24
- height >>= m_vChromaShift;25
- stride = reconPic->m_strideC;26
-27
- updateMD5Plane(m_frameEncoder->m_state[1], reconPic->getCbAddr(cuAddr), width, height, stride);28
- updateMD5Plane(m_frameEncoder->m_state[2], reconPic->getCrAddr(cuAddr), width, height, stride);29
- }30
- }31
- else if (m_param->decodedPictureHashSEI == 2)32
- {33
- uint32_t height = m_parallelFilter[row].getCUHeight();34
- uint32_t width = reconPic->m_picWidth;35
- intptr_t stride = reconPic->m_stride;36
-37
- if (!row)38
- m_frameEncoder->m_crc[0] = 0xffff;39
-40
- updateCRC(reconPic->getLumaAddr(cuAddr), m_frameEncoder->m_crc[0], height, width, stride);41
- if (m_param->internalCsp != X265_CSP_I400)42
- {43
- width >>= m_hChromaShift;44
- height >>= m_vChromaShift;45
- stride = reconPic->m_strideC;46
- m_frameEncoder->m_crc[1] = m_frameEncoder->m_crc[2] = 0xffff;47
-48
- updateCRC(reconPic->getCbAddr(cuAddr), m_frameEncoder->m_crc[1], height, width, stride);49
- updateCRC(reconPic->getCrAddr(cuAddr), m_frameEncoder->m_crc[2], height, width, stride);50
- }51
- }52
- else if (m_param->decodedPictureHashSEI == 3)53
- {54
- uint32_t width = reconPic->m_picWidth;55
- uint32_t height = m_parallelFilter[row].getCUHeight();56
- intptr_t stride = reconPic->m_stride;57
- uint32_t cuHeight = m_param->maxCUSize;58
-59
- if (!row)60
- m_frameEncoder->m_checksum[0] = 0;61
-62
- updateChecksum(reconPic->m_picOrg[0], m_frameEncoder->m_checksum[0], height, width, stride, row, cuHeight);63
- if (m_param->internalCsp != X265_CSP_I400)64
- {65
- width >>= m_hChromaShift;66
- height >>= m_vChromaShift;67
- stride = reconPic->m_strideC;68
- cuHeight >>= m_vChromaShift;69
-70
- if (!row)71
- m_frameEncoder->m_checksum[1] = m_frameEncoder->m_checksum[2] = 0;72
-73
- updateChecksum(reconPic->m_picOrg[1], m_frameEncoder->m_checksum[1], height, width, stride, row, cuHeight);74
- updateChecksum(reconPic->m_picOrg[2], m_frameEncoder->m_checksum[2], height, width, stride, row, cuHeight);75
- }76
- }77
+ uint32_t height = m_parallelFilter[row].getCUHeight();78
+ m_frameEncoder->initDecodedPictureHashSEI(row, cuAddr, height);79
} // end of (m_param->maxSlices == 1)80
81
if (ATOMIC_INC(&m_frameEncoder->m_completionCount) == 2 * (int)m_frameEncoder->m_numRows)82
x265_2.7.tar.gz/source/encoder/ratecontrol.cpp -> x265_2.9.tar.gz/source/encoder/ratecontrol.cpp
Changed
114
1
2
m_predictedBits = m_totalBits;3
updateVbvPlan(enc);4
rce->bufferFill = m_bufferFill;5
+ rce->vbvEndAdj = false;6
+ if (m_param->vbvBufferEnd && rce->encodeOrder >= m_param->vbvEndFrameAdjust * m_param->totalFrames)7
+ {8
+ rce->vbvEndAdj = true;9
+ rce->targetFill = 0;10
+ }11
12
int mincr = enc->m_vps.ptl.minCrForLevel;13
/* Profiles above Main10 don't require maxAU size check, so just set the maximum to a large value. */14
15
else16
{17
/* The spec has a special case for the first frame. */18
- if (rce->encodeOrder == 0)19
+ if (curFrame->m_lowres.bKeyframe)20
{21
/* 1.5 * (Max( PicSizeInSamplesY, fR * MaxLumaSr) + MaxLumaSr * (AuCpbRemovalTime[ 0 ] -AuNominalRemovalTime[ 0 ])) ? MinCr */22
double fr = 1. / 300;23
24
/* 1.5 * MaxLumaSr * (AuCpbRemovalTime[ n ] - AuCpbRemovalTime[ n - 1 ]) / MinCr */25
rce->frameSizeMaximum = 8 * 1.5 * enc->m_vps.ptl.maxLumaSrForLevel * m_frameDuration / mincr;26
}27
+ rce->frameSizeMaximum *= m_param->maxAUSizeFactor;28
}29
}30
if (!m_isAbr && m_2pass && m_param->rc.rateControlMode == X265_RC_CRF)31
32
curBits = predictSize(&m_pred[predType], frameQ[type], (double)satd);33
bufferFillCur -= curBits;34
}35
- if (m_param->vbvBufferEnd && rce->encodeOrder >= m_param->vbvEndFrameAdjust * m_param->totalFrames)36
+ if (rce->vbvEndAdj)37
{38
bool loopBreak = false;39
double bufferDiff = m_param->vbvBufferEnd - (m_bufferFill / m_bufferSize);40
- targetFill = m_bufferFill + m_bufferSize * (bufferDiff / (m_param->totalFrames - rce->encodeOrder));41
- if (bufferFillCur < targetFill)42
+ rce->targetFill = m_bufferFill + m_bufferSize * (bufferDiff / (m_param->totalFrames - rce->encodeOrder));43
+ if (bufferFillCur < rce->targetFill)44
{45
q *= 1.01;46
loopTerminate |= 1;47
48
double rcTol = bufferLeftPlanned / m_param->frameNumThreads * m_rateTolerance;49
int32_t encodedBitsSoFar = 0;50
double accFrameBits = predictRowsSizeSum(curFrame, rce, qpVbv, encodedBitsSoFar);51
+ double vbvEndBias = 0.95;52
53
/* * Don't increase the row QPs until a sufficent amount of the bits of54
* the frame have been processed, in case a flat area at the top of the55
56
while (qpVbv < qpMax57
&& (((accFrameBits > rce->frameSizePlanned + rcTol) ||58
(rce->bufferFill - accFrameBits < bufferLeftPlanned * 0.5) ||59
- (accFrameBits > rce->frameSizePlanned && qpVbv < rce->qpNoVbv))60
+ (accFrameBits > rce->frameSizePlanned && qpVbv < rce->qpNoVbv) ||61
+ (rce->vbvEndAdj && ((rce->bufferFill - accFrameBits) < (rce->targetFill * vbvEndBias))))62
&& (!m_param->rc.bStrictCbr ? 1 : abrOvershoot > 0.1)))63
{64
qpVbv += stepSize;65
66
while (qpVbv > qpMin67
&& (qpVbv > curEncData.m_rowStat[0].rowQp || m_singleFrameVbv)68
&& (((accFrameBits < rce->frameSizePlanned * 0.8f && qpVbv <= prevRowQp)69
- || accFrameBits < (rce->bufferFill - m_bufferSize + m_bufferRate) * 1.1)70
+ || accFrameBits < (rce->bufferFill - m_bufferSize + m_bufferRate) * 1.171
+ || (rce->vbvEndAdj && ((rce->bufferFill - accFrameBits) > (rce->targetFill * vbvEndBias))))72
&& (!m_param->rc.bStrictCbr ? 1 : abrOvershoot < 0)))73
{74
qpVbv -= stepSize;75
76
FrameData& curEncData = *curFrame->m_encData;77
int64_t actualBits = bits;78
Slice *slice = curEncData.m_slice;79
+ bool bEnableDistOffset = m_param->analysisMultiPassDistortion && m_param->rc.bStatRead;80
81
- if (m_param->rc.aqMode || m_isVbv || m_param->bAQMotion)82
+ if (m_param->rc.aqMode || m_isVbv || m_param->bAQMotion || bEnableDistOffset)83
{84
if (m_isVbv && !(m_2pass && m_param->rc.rateControlMode == X265_RC_CRF))85
{86
87
rce->qpaRc = curEncData.m_avgQpRc;88
}89
90
- if (m_param->rc.aqMode || m_param->bAQMotion)91
+ if (m_param->rc.aqMode || m_param->bAQMotion || bEnableDistOffset)92
{93
double avgQpAq = 0;94
- /* determine actual avg encoded QP, after AQ/cutree adjustments */95
+ /* determine actual avg encoded QP, after AQ/cutree/distortion adjustments */96
for (uint32_t i = 0; i < slice->m_sps->numCuInHeight; i++)97
avgQpAq += curEncData.m_rowStat[i].sumQpAq;98
99
100
/* called to write out the rate control frame stats info in multipass encodes */101
int RateControl::writeRateControlFrameStats(Frame* curFrame, RateControlEntry* rce)102
{103
- FrameData& curEncData = *curFrame->m_encData;104
- int ncu;105
- if (m_param->rc.qgSize == 8)106
- ncu = m_ncu * 4;107
- else108
- ncu = m_ncu;109
+ FrameData& curEncData = *curFrame->m_encData; 110
+ int ncu = (m_param->rc.qgSize == 8) ? m_ncu * 4 : m_ncu;111
char cType = rce->sliceType == I_SLICE ? (curFrame->m_lowres.sliceType == X265_TYPE_IDR ? 'I' : 'i')112
: rce->sliceType == P_SLICE ? 'P'113
: IS_REFERENCED(curFrame) ? 'B' : 'b';114
x265_2.7.tar.gz/source/encoder/ratecontrol.h -> x265_2.9.tar.gz/source/encoder/ratecontrol.h
Changed
10
1
2
double rowCplxrSum;3
double qpNoVbv;4
double bufferFill;5
+ double targetFill;6
+ bool vbvEndAdj;7
double frameDuration;8
double clippedDuration;9
double frameSizeEstimated; /* hold frameSize, updated from cu level vbv rc */10
x265_2.7.tar.gz/source/encoder/reference.cpp -> x265_2.9.tar.gz/source/encoder/reference.cpp
Changed
24
1
2
cuHeight >>= reconPic->m_vChromaShift;3
}4
5
- if (wp[c].bPresentFlag)6
+ if (wp[c].wtPresent)7
{8
if (!weightBuffer[c])9
{10
11
12
const pixel* src = reconPic->m_picOrg[c] + numWeightedRows * cuHeight * stride;13
pixel* dst = fpelPlane[c] + numWeightedRows * cuHeight * stride;14
-15
// Computing weighted CU rows16
int correction = IF_INTERNAL_PREC - X265_DEPTH; // intermediate interpolation depth17
- int padwidth = (width + 15) & ~15; // weightp assembly needs even 16 byte widths18
+ int padwidth = (width + 31) & ~31; // weightp assembly needs even 32 byte widths19
primitives.weight_pp(src, dst, stride, padwidth, height, w[c].weight, w[c].round << correction, w[c].shift + correction, w[c].offset);20
-21
// Extending Left & Right22
primitives.extendRowBorder(dst, stride, width, height, marginX);23
24
x265_2.7.tar.gz/source/encoder/search.cpp -> x265_2.9.tar.gz/source/encoder/search.cpp
Changed
409
1
2
m_me.init(param.internalCsp);3
4
bool ok = m_quant.init(param.psyRdoq, scalingList, m_entropyCoder);5
- if (m_param->noiseReductionIntra || m_param->noiseReductionInter || m_param->rc.vbvBufferSize)6
+ if (m_param->noiseReductionIntra || m_param->noiseReductionInter )7
ok &= m_quant.allocNoiseReduction(param);8
9
ok &= Predict::allocBuffers(param.internalCsp); /* sets m_hChromaShift & m_vChromaShift */10
11
// store original entropy coding status12
if (bEnableRDOQ)13
m_entropyCoder.estBit(m_entropyCoder.m_estBitsSbac, log2TrSize, true);14
-15
- primitives.cu[sizeIdx].calcresidual(fenc, pred, residual, stride);16
+ primitives.cu[sizeIdx].calcresidual[stride % 64 == 0](fenc, pred, residual, stride);17
18
uint32_t numSig = m_quant.transformNxN(cu, fenc, stride, residual, stride, coeffY, log2TrSize, TEXT_LUMA, absPartIdx, false);19
if (numSig)20
{21
m_quant.invtransformNxN(cu, residual, stride, coeffY, log2TrSize, TEXT_LUMA, true, false, numSig);22
- primitives.cu[sizeIdx].add_ps(reconQt, reconQtStride, pred, residual, stride, stride);23
+ bool reconQtYuvAlign = m_rqt[qtLayer].reconQtYuv.getAddrOffset(absPartIdx, mode.predYuv.m_size) % 64 == 0;24
+ bool predAlign = mode.predYuv.getAddrOffset(absPartIdx, mode.predYuv.m_size) % 64 == 0;25
+ bool residualAlign = m_rqt[cuGeom.depth].tmpResiYuv.getAddrOffset(absPartIdx, mode.predYuv.m_size) % 64 == 0;26
+ bool bufferAlignCheck = (reconQtStride % 64 == 0) && (stride % 64 == 0) && reconQtYuvAlign && predAlign && residualAlign;27
+ primitives.cu[sizeIdx].add_ps[bufferAlignCheck](reconQt, reconQtStride, pred, residual, stride, stride);28
}29
else30
// no coded residual, recon = pred31
32
33
coeff_t* coeff = (useTSkip ? m_tsCoeff : coeffY);34
pixel* tmpRecon = (useTSkip ? m_tsRecon : reconQt);35
+ bool tmpReconAlign = (useTSkip ? 1 : (m_rqt[qtLayer].reconQtYuv.getAddrOffset(absPartIdx, m_rqt[qtLayer].reconQtYuv.m_size) % 64 == 0));36
uint32_t tmpReconStride = (useTSkip ? MAX_TS_SIZE : reconQtStride);37
38
- primitives.cu[sizeIdx].calcresidual(fenc, pred, residual, stride);39
+ primitives.cu[sizeIdx].calcresidual[stride % 64 == 0](fenc, pred, residual, stride);40
41
uint32_t numSig = m_quant.transformNxN(cu, fenc, stride, residual, stride, coeff, log2TrSize, TEXT_LUMA, absPartIdx, useTSkip);42
if (numSig)43
{44
m_quant.invtransformNxN(cu, residual, stride, coeff, log2TrSize, TEXT_LUMA, true, useTSkip, numSig);45
- primitives.cu[sizeIdx].add_ps(tmpRecon, tmpReconStride, pred, residual, stride, stride);46
+ bool residualAlign = m_rqt[cuGeom.depth].tmpResiYuv.getAddrOffset(absPartIdx, m_rqt[cuGeom.depth].tmpResiYuv.m_size) % 64 == 0;47
+ bool predAlign = predYuv->getAddrOffset(absPartIdx, predYuv->m_size) % 64 == 0;48
+ bool bufferAlignCheck = (stride % 64 == 0) && (tmpReconStride % 64 == 0) && tmpReconAlign && residualAlign && predAlign;49
+ primitives.cu[sizeIdx].add_ps[bufferAlignCheck](tmpRecon, tmpReconStride, pred, residual, stride, stride);50
}51
else if (useTSkip)52
{53
54
coeff_t* coeffY = cu.m_trCoeff[0] + coeffOffsetY;55
56
uint32_t sizeIdx = log2TrSize - 2;57
- primitives.cu[sizeIdx].calcresidual(fenc, pred, residual, stride);58
+ primitives.cu[sizeIdx].calcresidual[stride % 64 == 0](fenc, pred, residual, stride);59
60
PicYuv* reconPic = m_frame->m_reconPic;61
pixel* picReconY = reconPic->getLumaAddr(cu.m_cuAddr, cuGeom.absPartIdx + absPartIdx);62
63
if (numSig)64
{65
m_quant.invtransformNxN(cu, residual, stride, coeffY, log2TrSize, TEXT_LUMA, true, false, numSig);66
- primitives.cu[sizeIdx].add_ps(picReconY, picStride, pred, residual, stride, stride);67
+ bool picReconYAlign = (reconPic->m_cuOffsetY[cu.m_cuAddr] + reconPic->m_buOffsetY[cuGeom.absPartIdx + absPartIdx]) % 64 == 0;68
+ bool predAlign = mode.predYuv.getAddrOffset(absPartIdx, mode.predYuv.m_size) % 64 == 0;69
+ bool residualAlign = m_rqt[cuGeom.depth].tmpResiYuv.getAddrOffset(absPartIdx, m_rqt[cuGeom.depth].tmpResiYuv.m_size)% 64 == 0;70
+ bool bufferAlignCheck = (picStride % 64 == 0) && (stride % 64 == 0) && picReconYAlign && predAlign && residualAlign;71
+ primitives.cu[sizeIdx].add_ps[bufferAlignCheck](picReconY, picStride, pred, residual, stride, stride);72
cu.setCbfSubParts(1 << tuDepth, TEXT_LUMA, absPartIdx, fullDepth);73
}74
else75
76
predIntraChromaAng(chromaPredMode, pred, stride, log2TrSizeC);77
cu.setTransformSkipPartRange(0, ttype, absPartIdxC, tuIterator.absPartIdxStep);78
79
- primitives.cu[sizeIdxC].calcresidual(fenc, pred, residual, stride);80
+ primitives.cu[sizeIdxC].calcresidual[stride % 64 == 0](fenc, pred, residual, stride);81
+82
uint32_t numSig = m_quant.transformNxN(cu, fenc, stride, residual, stride, coeffC, log2TrSizeC, ttype, absPartIdxC, false);83
if (numSig)84
{85
m_quant.invtransformNxN(cu, residual, stride, coeffC, log2TrSizeC, ttype, true, false, numSig);86
- primitives.cu[sizeIdxC].add_ps(reconQt, reconQtStride, pred, residual, stride, stride);87
+ bool reconQtAlign = m_rqt[qtLayer].reconQtYuv.getChromaAddrOffset(absPartIdxC) % 64 == 0;88
+ bool predAlign = mode.predYuv.getChromaAddrOffset(absPartIdxC) % 64 == 0;89
+ bool residualAlign = resiYuv.getChromaAddrOffset(absPartIdxC) % 64 == 0;90
+ bool bufferAlignCheck = reconQtAlign && predAlign && residualAlign && (reconQtStride % 64 == 0) && (stride % 64 == 0);91
+ primitives.cu[sizeIdxC].add_ps[bufferAlignCheck](reconQt, reconQtStride, pred, residual, stride, stride);92
cu.setCbfPartRange(1 << tuDepth, ttype, absPartIdxC, tuIterator.absPartIdxStep);93
}94
else95
96
pixel* recon = (useTSkip ? m_tsRecon : reconQt);97
uint32_t reconStride = (useTSkip ? MAX_TS_SIZE : reconQtStride);98
99
- primitives.cu[sizeIdxC].calcresidual(fenc, pred, residual, stride);100
+ primitives.cu[sizeIdxC].calcresidual[stride % 64 == 0](fenc, pred, residual, stride);101
102
uint32_t numSig = m_quant.transformNxN(cu, fenc, stride, residual, stride, coeff, log2TrSizeC, ttype, absPartIdxC, useTSkip);103
if (numSig)104
{105
m_quant.invtransformNxN(cu, residual, stride, coeff, log2TrSizeC, ttype, true, useTSkip, numSig);106
- primitives.cu[sizeIdxC].add_ps(recon, reconStride, pred, residual, stride, stride);107
+ bool reconAlign = (useTSkip ? 1 : m_rqt[qtLayer].reconQtYuv.getChromaAddrOffset(absPartIdxC)) % 64 == 0;108
+ bool predYuvAlign = mode.predYuv.getChromaAddrOffset(absPartIdxC) % 64 == 0;109
+ bool residualAlign = m_rqt[cuGeom.depth].tmpResiYuv.getChromaAddrOffset(absPartIdxC) % 64 == 0;110
+ bool bufferAlignCheck = reconAlign && predYuvAlign && residualAlign && (reconStride % 64 == 0) && (stride % 64 == 0);111
+ primitives.cu[sizeIdxC].add_ps[bufferAlignCheck](recon, reconStride, pred, residual, stride, stride);112
cu.setCbfPartRange(1 << tuDepth, ttype, absPartIdxC, tuIterator.absPartIdxStep);113
}114
else if (useTSkip)115
116
117
X265_CHECK(!cu.m_transformSkip[ttype][0], "transform skip not supported at low RD levels\n");118
119
- primitives.cu[sizeIdxC].calcresidual(fenc, pred, residual, stride);120
+ primitives.cu[sizeIdxC].calcresidual[stride % 64 == 0](fenc, pred, residual, stride);121
+122
uint32_t numSig = m_quant.transformNxN(cu, fenc, stride, residual, stride, coeffC, log2TrSizeC, ttype, absPartIdxC, false);123
if (numSig)124
{125
m_quant.invtransformNxN(cu, residual, stride, coeffC, log2TrSizeC, ttype, true, false, numSig);126
- primitives.cu[sizeIdxC].add_ps(picReconC, picStride, pred, residual, stride, stride);127
+ bool picReconCAlign = (reconPic->m_cuOffsetC[cu.m_cuAddr] + reconPic->m_buOffsetC[cuGeom.absPartIdx + absPartIdxC]) % 64 == 0;128
+ bool predAlign = mode.predYuv.getChromaAddrOffset(absPartIdxC) % 64 == 0;129
+ bool residualAlign = resiYuv.getChromaAddrOffset(absPartIdxC)% 64 == 0;130
+ bool bufferAlignCheck = picReconCAlign && predAlign && residualAlign && (picStride % 64 == 0) && (stride % 64 == 0);131
+ primitives.cu[sizeIdxC].add_ps[bufferAlignCheck](picReconC, picStride, pred, residual, stride, stride);132
cu.setCbfPartRange(1 << tuDepth, ttype, absPartIdxC, tuIterator.absPartIdxStep);133
}134
else135
136
137
pixel nScale[129];138
intraNeighbourBuf[1][0] = intraNeighbourBuf[0][0];139
- primitives.scale1D_128to64(nScale + 1, intraNeighbourBuf[0] + 1);140
+ primitives.scale1D_128to64[NONALIGNED](nScale + 1, intraNeighbourBuf[0] + 1);141
142
// we do not estimate filtering for downscaled samples143
memcpy(&intraNeighbourBuf[0][1], &nScale[1], 2 * 64 * sizeof(pixel)); // Top & Left pixels144
145
bestME[list].mvCost = mvCost;146
}147
}148
-149
-void Search::searchMV(Mode& interMode, const PredictionUnit& pu, int list, int ref, MV& outmv)150
+void Search::searchMV(Mode& interMode, const PredictionUnit& pu, int list, int ref, MV& outmv, MV mvp, int numMvc, MV* mvc)151
{152
CUData& cu = interMode.cu;153
const Slice *slice = m_slice;154
- MV mv = cu.m_mv[list][pu.puAbsPartIdx];155
+ MV mv;156
+ if (m_param->interRefine == 1)157
+ mv = mvp;158
+ else159
+ mv = cu.m_mv[list][pu.puAbsPartIdx];160
cu.clipMv(mv);161
MV mvmin, mvmax;162
setSearchRange(cu, mv, m_param->searchRange, mvmin, mvmax);163
- m_me.refineMV(&slice->m_mref[list][ref], mvmin, mvmax, mv, outmv);164
+ if (m_param->interRefine == 1)165
+ m_me.motionEstimate(&m_slice->m_mref[list][ref], mvmin, mvmax, mv, numMvc, mvc, m_param->searchRange, outmv, m_param->maxSlices,166
+ m_param->bSourceReferenceEstimation ? m_slice->m_refFrameList[list][ref]->m_fencPic->getLumaAddr(0) : 0);167
+ else168
+ m_me.refineMV(&slice->m_mref[list][ref], mvmin, mvmax, mv, outmv);169
}170
-171
/* find the best inter prediction for each PU of specified mode */172
void Search::predInterSearch(Mode& interMode, const CUGeom& cuGeom, bool bChromaMC, uint32_t refMasks[2])173
{174
175
int totalmebits = 0;176
MV mvzero(0, 0);177
Yuv& tmpPredYuv = m_rqt[cuGeom.depth].tmpPredYuv;178
-179
MergeData merge;180
memset(&merge, 0, sizeof(merge));181
-182
+ bool useAsMVP = false;183
for (int puIdx = 0; puIdx < numPart; puIdx++)184
{185
MotionData* bestME = interMode.bestME[puIdx];186
PredictionUnit pu(cu, cuGeom, puIdx);187
-188
m_me.setSourcePU(*interMode.fencYuv, pu.ctuAddr, pu.cuAbsPartIdx, pu.puAbsPartIdx, pu.width, pu.height, m_param->searchMethod, m_param->subpelRefine, bChromaMC);189
-190
+ useAsMVP = false;191
+ x265_analysis_inter_data* interDataCTU = NULL;192
+ int cuIdx;193
+ cuIdx = (interMode.cu.m_cuAddr * m_param->num4x4Partitions) + cuGeom.absPartIdx;194
+ if (m_param->analysisReuseLevel == 10 && m_param->interRefine > 1)195
+ {196
+ interDataCTU = m_frame->m_analysisData.interData;197
+ if ((cu.m_predMode[pu.puAbsPartIdx] == interDataCTU->modes[cuIdx + pu.puAbsPartIdx])198
+ && (cu.m_partSize[pu.puAbsPartIdx] == interDataCTU->partSize[cuIdx + pu.puAbsPartIdx])199
+ && !(interDataCTU->mergeFlag[cuIdx + puIdx])200
+ && (cu.m_cuDepth[0] == interDataCTU->depth[cuIdx]))201
+ useAsMVP = true;202
+ }203
/* find best cost merge candidate. note: 2Nx2N merge and bidir are handled as separate modes */204
uint32_t mrgCost = numPart == 1 ? MAX_UINT : mergeEstimation(cu, cuGeom, pu, puIdx, merge);205
-206
bestME[0].cost = MAX_UINT;207
bestME[1].cost = MAX_UINT;208
209
210
bool bDoUnidir = true;211
212
cu.getNeighbourMV(puIdx, pu.puAbsPartIdx, interMode.interNeighbours);213
-214
/* Uni-directional prediction */215
if ((m_param->analysisLoad && m_param->analysisReuseLevel > 1 && m_param->analysisReuseLevel != 10)216
- || (m_param->analysisMultiPassRefine && m_param->rc.bStatRead) || (m_param->bMVType == AVC_INFO))217
+ || (m_param->analysisMultiPassRefine && m_param->rc.bStatRead) || (m_param->bMVType == AVC_INFO) || (useAsMVP))218
{219
for (int list = 0; list < numPredDir; list++)220
{221
- int ref = bestME[list].ref;222
+223
+ int ref = -1;224
+ if (useAsMVP)225
+ ref = interDataCTU->refIdx[list][cuIdx + puIdx];226
+227
+ else228
+ ref = bestME[list].ref;229
if (ref < 0)230
+ {231
continue;232
-233
+ }234
uint32_t bits = m_listSelBits[list] + MVP_IDX_BITS;235
bits += getTUBits(ref, numRefIdx[list]);236
237
int numMvc = cu.getPMV(interMode.interNeighbours, list, ref, interMode.amvpCand[list][ref], mvc);238
-239
const MV* amvp = interMode.amvpCand[list][ref];240
int mvpIdx = selectMVP(cu, pu, amvp, list, ref);241
- MV mvmin, mvmax, outmv, mvp = amvp[mvpIdx];242
-243
+ MV mvmin, mvmax, outmv, mvp;244
+ if (useAsMVP)245
+ {246
+ mvp = interDataCTU->mv[list][cuIdx + puIdx].word;247
+ mvpIdx = interDataCTU->mvpIdx[list][cuIdx + puIdx];248
+ }249
+ else250
+ mvp = amvp[mvpIdx];251
if (m_param->searchMethod == X265_SEA)252
{253
int puX = puIdx & 1;254
255
bits += m_me.bitcost(outmv);256
uint32_t mvCost = m_me.mvcost(outmv);257
uint32_t cost = (satdCost - mvCost) + m_rdCost.getCost(bits);258
-259
/* Refine MVP selection, updates: mvpIdx, bits, cost */260
- if (!m_param->analysisMultiPassRefine)261
+ if (!(m_param->analysisMultiPassRefine || useAsMVP))262
mvp = checkBestMVP(amvp, outmv, mvpIdx, bits, cost);263
else264
{265
266
bestME[list].cost = cost;267
bestME[list].bits = bits;268
bestME[list].mvCost = mvCost;269
+ bestME[list].ref = ref;270
}271
bDoUnidir = false;272
} 273
274
/* Generate reference subpels */275
predInterLumaPixel(pu, bidirYuv[0], *refPic0, bestME[0].mv);276
predInterLumaPixel(pu, bidirYuv[1], *refPic1, bestME[1].mv);277
-278
- primitives.pu[m_me.partEnum].pixelavg_pp(tmpPredYuv.m_buf[0], tmpPredYuv.m_size, bidirYuv[0].getLumaAddr(pu.puAbsPartIdx), bidirYuv[0].m_size,279
+ primitives.pu[m_me.partEnum].pixelavg_pp[(tmpPredYuv.m_size % 64 == 0) && (bidirYuv[0].m_size % 64 == 0) && (bidirYuv[1].m_size % 64 == 0)](tmpPredYuv.m_buf[0], tmpPredYuv.m_size, bidirYuv[0].getLumaAddr(pu.puAbsPartIdx), bidirYuv[0].m_size,280
bidirYuv[1].getLumaAddr(pu.puAbsPartIdx), bidirYuv[1].m_size, 32);281
satdCost = m_me.bufSATD(tmpPredYuv.m_buf[0], tmpPredYuv.m_size);282
}283
284
const pixel* ref0 = m_slice->m_mref[0][bestME[0].ref].getLumaAddr(pu.ctuAddr, pu.cuAbsPartIdx + pu.puAbsPartIdx);285
const pixel* ref1 = m_slice->m_mref[1][bestME[1].ref].getLumaAddr(pu.ctuAddr, pu.cuAbsPartIdx + pu.puAbsPartIdx);286
intptr_t refStride = slice->m_mref[0][0].lumaStride;287
-288
- primitives.pu[m_me.partEnum].pixelavg_pp(tmpPredYuv.m_buf[0], tmpPredYuv.m_size, ref0, refStride, ref1, refStride, 32);289
+ primitives.pu[m_me.partEnum].pixelavg_pp[(tmpPredYuv.m_size % 64 == 0) && (refStride % 64 == 0)](tmpPredYuv.m_buf[0], tmpPredYuv.m_size, ref0, refStride, ref1, refStride, 32);290
satdCost = m_me.bufSATD(tmpPredYuv.m_buf[0], tmpPredYuv.m_size);291
}292
-293
MV mvp0 = bestME[0].mvp;294
int mvpIdx0 = bestME[0].mvpIdx;295
uint32_t bits0 = bestME[0].bits - m_me.bitcost(bestME[0].mv, mvp0) + m_me.bitcost(mvzero, mvp0);296
297
}298
else299
{300
- primitives.cu[sizeIdx].blockfill_s(curResiY, strideResiY, 0);301
+ primitives.cu[sizeIdx].blockfill_s[strideResiY % 64 == 0](curResiY, strideResiY, 0);302
cu.setCbfSubParts(0, TEXT_LUMA, absPartIdx, depth);303
}304
305
306
}307
else308
{309
- primitives.cu[sizeIdxC].blockfill_s(curResiU, strideResiC, 0);310
+ primitives.cu[sizeIdxC].blockfill_s[strideResiC % 64 == 0](curResiU, strideResiC, 0);311
cu.setCbfPartRange(0, TEXT_CHROMA_U, absPartIdxC, tuIterator.absPartIdxStep);312
}313
314
315
}316
else317
{318
- primitives.cu[sizeIdxC].blockfill_s(curResiV, strideResiC, 0);319
+ primitives.cu[sizeIdxC].blockfill_s[strideResiC % 64 == 0](curResiV, strideResiC, 0);320
cu.setCbfPartRange(0, TEXT_CHROMA_V, absPartIdxC, tuIterator.absPartIdxStep);321
}322
}323
324
// non-zero cost calculation for luma - This is an approximation325
// finally we have to encode correct cbf after comparing with null cost326
pixel* curReconY = m_rqt[qtLayer].reconQtYuv.getLumaAddr(absPartIdx);327
+ bool curReconYAlign = m_rqt[qtLayer].reconQtYuv.getAddrOffset(absPartIdx, m_rqt[qtLayer].reconQtYuv.m_size) % 64 == 0;328
uint32_t strideReconY = m_rqt[qtLayer].reconQtYuv.m_size;329
- primitives.cu[partSize].add_ps(curReconY, strideReconY, mode.predYuv.getLumaAddr(absPartIdx), curResiY, mode.predYuv.m_size, strideResiY);330
+ bool predYuvAlign = mode.predYuv.getAddrOffset(absPartIdx, mode.predYuv.m_size) % 64 == 0;331
+ bool curResiYAlign = m_rqt[qtLayer].resiQtYuv.getAddrOffset(absPartIdx, m_rqt[qtLayer].resiQtYuv.m_size) % 64 == 0;332
+ bool bufferAlignCheck = curReconYAlign && predYuvAlign && curResiYAlign && (strideReconY % 64 == 0) && (mode.predYuv.m_size % 64 == 0) && (strideResiY % 64 == 0);333
+ primitives.cu[partSize].add_ps[bufferAlignCheck](curReconY, strideReconY, mode.predYuv.getLumaAddr(absPartIdx), curResiY, mode.predYuv.m_size, strideResiY);334
335
const sse_t nonZeroDistY = primitives.cu[partSize].sse_pp(fenc, fencYuv->m_size, curReconY, strideReconY);336
uint32_t nzCbfBitsY = m_entropyCoder.estimateCbfBits(cbfFlag[TEXT_LUMA][0], TEXT_LUMA, tuDepth);337
338
{339
cbfFlag[TEXT_LUMA][0] = 0;340
singleBits[TEXT_LUMA][0] = 0;341
- primitives.cu[partSize].blockfill_s(curResiY, strideResiY, 0);342
+ primitives.cu[partSize].blockfill_s[strideResiY % 64 == 0](curResiY, strideResiY, 0);343
#if CHECKED_BUILD || _DEBUG344
uint32_t numCoeffY = 1 << (log2TrSize << 1);345
memset(coeffCurY, 0, sizeof(coeff_t)* numCoeffY);346
347
{348
if (checkTransformSkipY)349
minCost[TEXT_LUMA][0] = estimateNullCbfCost(zeroDistY, zeroEnergyY, tuDepth, TEXT_LUMA);350
- primitives.cu[partSize].blockfill_s(curResiY, strideResiY, 0);351
+ primitives.cu[partSize].blockfill_s[strideResiY % 64 == 0](curResiY, strideResiY, 0);352
singleDist[TEXT_LUMA][0] = zeroDistY;353
singleBits[TEXT_LUMA][0] = 0;354
singleEnergy[TEXT_LUMA][0] = zeroEnergyY;355
356
// finally we have to encode correct cbf after comparing with null cost357
pixel* curReconC = m_rqt[qtLayer].reconQtYuv.getChromaAddr(chromaId, absPartIdxC);358
uint32_t strideReconC = m_rqt[qtLayer].reconQtYuv.m_csize;359
- primitives.cu[partSizeC].add_ps(curReconC, strideReconC, mode.predYuv.getChromaAddr(chromaId, absPartIdxC), curResiC, mode.predYuv.m_csize, strideResiC);360
+ bool curReconCAlign = m_rqt[qtLayer].reconQtYuv.getChromaAddrOffset(absPartIdxC) % 64 == 0;361
+ bool predYuvAlign = mode.predYuv.getChromaAddrOffset(absPartIdxC) % 64 == 0;362
+ bool curResiCAlign = m_rqt[qtLayer].resiQtYuv.getChromaAddrOffset(absPartIdxC) % 64 == 0;363
+ bool bufferAlignCheck = curReconCAlign && predYuvAlign && curResiCAlign && (strideReconC % 64 == 0) && (mode.predYuv.m_csize % 64 == 0) && (strideResiC % 64 == 0);364
+ primitives.cu[partSizeC].add_ps[bufferAlignCheck](curReconC, strideReconC, mode.predYuv.getChromaAddr(chromaId, absPartIdxC), curResiC, mode.predYuv.m_csize, strideResiC);365
sse_t nonZeroDistC = m_rdCost.scaleChromaDist(chromaId, primitives.cu[partSizeC].sse_pp(fenc, fencYuv->m_csize, curReconC, strideReconC));366
uint32_t nzCbfBitsC = m_entropyCoder.estimateCbfBits(cbfFlag[chromaId][tuIterator.section], (TextType)chromaId, tuDepth);367
uint32_t nonZeroEnergyC = 0; uint64_t singleCostC = 0;368
369
{370
cbfFlag[chromaId][tuIterator.section] = 0;371
singleBits[chromaId][tuIterator.section] = 0;372
- primitives.cu[partSizeC].blockfill_s(curResiC, strideResiC, 0);373
+ primitives.cu[partSizeC].blockfill_s[strideResiC % 64 == 0](curResiC, strideResiC, 0);374
#if CHECKED_BUILD || _DEBUG375
uint32_t numCoeffC = 1 << (log2TrSizeC << 1);376
memset(coeffCurC + subTUOffset, 0, sizeof(coeff_t) * numCoeffC);377
378
{379
if (checkTransformSkipC)380
minCost[chromaId][tuIterator.section] = estimateNullCbfCost(zeroDistC, zeroEnergyC, tuDepthC, (TextType)chromaId);381
- primitives.cu[partSizeC].blockfill_s(curResiC, strideResiC, 0);382
+ primitives.cu[partSizeC].blockfill_s[strideResiC % 64 == 0](curResiC, strideResiC, 0);383
singleBits[chromaId][tuIterator.section] = 0;384
singleDist[chromaId][tuIterator.section] = zeroDistC;385
singleEnergy[chromaId][tuIterator.section] = zeroEnergyC;386
387
const uint32_t skipSingleBitsY = m_entropyCoder.getNumberOfWrittenBits();388
389
m_quant.invtransformNxN(cu, m_tsResidual, trSize, m_tsCoeff, log2TrSize, TEXT_LUMA, false, true, numSigTSkipY);390
+ bool predYuvAlign = mode.predYuv.getAddrOffset(absPartIdx, mode.predYuv.m_size) % 64 == 0;391
392
- primitives.cu[partSize].add_ps(m_tsRecon, trSize, mode.predYuv.getLumaAddr(absPartIdx), m_tsResidual, mode.predYuv.m_size, trSize);393
+ bool bufferAlignCheck = predYuvAlign && (trSize % 64 == 0) && (mode.predYuv.m_size % 64 == 0);394
+ primitives.cu[partSize].add_ps[bufferAlignCheck](m_tsRecon, trSize, mode.predYuv.getLumaAddr(absPartIdx), m_tsResidual, mode.predYuv.m_size, trSize);395
nonZeroDistY = primitives.cu[partSize].sse_pp(fenc, fencYuv->m_size, m_tsRecon, trSize);396
397
if (m_rdCost.m_psyRd)398
399
400
m_quant.invtransformNxN(cu, m_tsResidual, trSizeC, m_tsCoeff,401
log2TrSizeC, (TextType)chromaId, false, true, numSigTSkipC);402
- primitives.cu[partSizeC].add_ps(m_tsRecon, trSizeC, mode.predYuv.getChromaAddr(chromaId, absPartIdxC), m_tsResidual, mode.predYuv.m_csize, trSizeC);403
+ bool predYuvAlign = mode.predYuv.getChromaAddrOffset(absPartIdxC) % 64 == 0;404
+ bool bufferAlignCheck = predYuvAlign && (trSizeC % 64 == 0) && (mode.predYuv.m_csize % 64 == 0) && (trSizeC % 64 == 0);405
+ primitives.cu[partSizeC].add_ps[bufferAlignCheck](m_tsRecon, trSizeC, mode.predYuv.getChromaAddr(chromaId, absPartIdxC), m_tsResidual, mode.predYuv.m_csize, trSizeC);406
nonZeroDistC = m_rdCost.scaleChromaDist(chromaId, primitives.cu[partSizeC].sse_pp(fenc, fencYuv->m_csize, m_tsRecon, trSizeC));407
if (m_rdCost.m_psyRd)408
{409
x265_2.7.tar.gz/source/encoder/search.h -> x265_2.9.tar.gz/source/encoder/search.h
Changed
11
1
2
3
// estimation inter prediction (non-skip)4
void predInterSearch(Mode& interMode, const CUGeom& cuGeom, bool bChromaMC, uint32_t masks[2]);5
-6
- void searchMV(Mode& interMode, const PredictionUnit& pu, int list, int ref, MV& outmv);7
+ void searchMV(Mode& interMode, const PredictionUnit& pu, int list, int ref, MV& outmv, MV mvp, int numMvc, MV* mvc);8
// encode residual and compute rd-cost for inter mode9
void encodeResAndCalcRdInterCU(Mode& interMode, const CUGeom& cuGeom);10
void encodeResAndCalcRdSkipCU(Mode& interMode);11
x265_2.7.tar.gz/source/encoder/sei.cpp -> x265_2.9.tar.gz/source/encoder/sei.cpp
Changed
141
1
2
};3
4
/* marshal a single SEI message sei, storing the marshalled representation5
- * in bitstream bs */6
-void SEI::write(Bitstream& bs, const SPS& sps)7
+* in bitstream bs */8
+void SEI::writeSEImessages(Bitstream& bs, const SPS& sps, NalUnitType nalUnitType, NALList& list, int isNested)9
{10
- uint32_t type = m_payloadType;11
+ if (!isNested)12
+ bs.resetBits();13
+14
+ BitCounter counter;15
+ m_bitIf = &counter;16
+ writeSEI(sps);17
+ /* count the size of the payload and return the size in bits */18
+ X265_CHECK(0 == (counter.getNumberOfWrittenBits() & 7), "payload unaligned\n");19
+ uint32_t payloadData = counter.getNumberOfWrittenBits() >> 3;20
+21
+ // set bitstream22
m_bitIf = &bs;23
- BitCounter count;24
- bool hrdTypes = (m_payloadType == ACTIVE_PARAMETER_SETS || m_payloadType == PICTURE_TIMING || m_payloadType == BUFFERING_PERIOD);25
- if (hrdTypes)26
- {27
- m_bitIf = &count;28
- /* virtual writeSEI method, write to bit counter to determine size */29
- writeSEI(sps);30
- m_bitIf = &bs;31
- uint32_t payloadType = m_payloadType;32
- for (; payloadType >= 0xff; payloadType -= 0xff)33
- WRITE_CODE(0xff, 8, "payload_type");34
- }35
- WRITE_CODE(type, 8, "payload_type");36
- uint32_t payloadSize;37
- if (hrdTypes || m_payloadType == USER_DATA_UNREGISTERED || m_payloadType == USER_DATA_REGISTERED_ITU_T_T35)38
+39
+ uint32_t payloadType = m_payloadType;40
+ for (; payloadType >= 0xff; payloadType -= 0xff)41
+ WRITE_CODE(0xff, 8, "payload_type");42
+ WRITE_CODE(payloadType, 8, "payload_type");43
+44
+ uint32_t payloadSize = payloadData;45
+ for (; payloadSize >= 0xff; payloadSize -= 0xff)46
+ WRITE_CODE(0xff, 8, "payload_size");47
+ WRITE_CODE(payloadSize, 8, "payload_size");48
+49
+ // virtual writeSEI method, write to bs 50
+ writeSEI(sps);51
+52
+ if (!isNested)53
{54
- if (hrdTypes)55
- {56
- X265_CHECK(0 == (count.getNumberOfWrittenBits() & 7), "payload unaligned\n");57
- payloadSize = count.getNumberOfWrittenBits() >> 3;58
- }59
- else if (m_payloadType == USER_DATA_UNREGISTERED)60
- payloadSize = m_payloadSize + 16;61
- else62
- payloadSize = m_payloadSize;63
-64
- for (; payloadSize >= 0xff; payloadSize -= 0xff)65
- WRITE_CODE(0xff, 8, "payload_size");66
- WRITE_CODE(payloadSize, 8, "payload_size");67
+ bs.writeByteAlignment();68
+ list.serialize(nalUnitType, bs);69
}70
- else71
- WRITE_CODE(m_payloadSize, 8, "payload_size");72
- /* virtual writeSEI method, write to bs */73
- writeSEI(sps);74
}75
76
void SEI::writeByteAlign()77
78
{79
m_payloadSize = size;80
}81
+82
+/* charSet = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/" */83
+84
+char* SEI::base64Decode(char encodedString[], int base64EncodeLength)85
+{86
+ char* decodedString;87
+ decodedString = (char*)malloc(sizeof(char) * ((base64EncodeLength / 4) * 3));88
+ int i, j, k = 0;89
+ // stores the bitstream90
+ int bitstream = 0;91
+ // countBits stores current number of bits in bitstream92
+ int countBits = 0;93
+ // selects 4 characters from encodedString at a time. Find the position of each encoded character in charSet and stores in bitstream94
+ for (i = 0; i < base64EncodeLength; i += 4)95
+ {96
+ bitstream = 0, countBits = 0;97
+ for (j = 0; j < 4; j++)98
+ {99
+ // make space for 6 bits100
+ if (encodedString[i + j] != '=')101
+ {102
+ bitstream = bitstream << 6;103
+ countBits += 6;104
+ }105
+ // Finding the position of each encoded character in charSet and storing in bitstream, use OR '|' operator to store bits106
+107
+ if (encodedString[i + j] >= 'A' && encodedString[i + j] <= 'Z')108
+ bitstream = bitstream | (encodedString[i + j] - 'A');109
+110
+ else if (encodedString[i + j] >= 'a' && encodedString[i + j] <= 'z')111
+ bitstream = bitstream | (encodedString[i + j] - 'a' + 26);112
+ 113
+ else if (encodedString[i + j] >= '0' && encodedString[i + j] <= '9')114
+ bitstream = bitstream | (encodedString[i + j] - '0' + 52);115
+ 116
+ // '+' occurs in 62nd position in charSet117
+ else if (encodedString[i + j] == '+')118
+ bitstream = bitstream | 62;119
+ 120
+ // '/' occurs in 63rd position in charSet121
+ else if (encodedString[i + j] == '/')122
+ bitstream = bitstream | 63;123
+ 124
+ // to delete appended bits during encoding125
+ else126
+ {127
+ bitstream = bitstream >> 2;128
+ countBits -= 2;129
+ }130
+ }131
+ 132
+ while (countBits != 0)133
+ {134
+ countBits -= 8;135
+ decodedString[k++] = (bitstream >> countBits) & 255;136
+ }137
+ }138
+ return decodedString;139
+}140
+141
x265_2.7.tar.gz/source/encoder/sei.h -> x265_2.9.tar.gz/source/encoder/sei.h
Changed
135
1
2
#include "common.h"3
#include "bitstream.h"4
#include "slice.h"5
+#include "nal.h"6
+#include "md5.h"7
8
namespace X265_NS {9
// private namespace10
11
class SEI : public SyntaxElementWriter12
{13
public:14
- /* SEI users call write() to marshal an SEI to a bitstream.15
- * The write() method calls writeSEI() which encodes the header */16
- void write(Bitstream& bs, const SPS& sps);17
-18
+ /* SEI users call writeSEImessages() to marshal an SEI to a bitstream.19
+ * The writeSEImessages() method calls writeSEI() which encodes the header */20
+ void writeSEImessages(Bitstream& bs, const SPS& sps, NalUnitType nalUnitType, NALList& list, int isNested);21
void setSize(uint32_t size);22
+ static char* base64Decode(char encodedString[], int base64EncodeLength);23
virtual ~SEI() {}24
protected:25
SEIPayloadType m_payloadType;26
27
void writeByteAlign();28
};29
30
+//seongnam.oh@samsung.com :: for the Creative Intent Meta Data Encoding31
+class SEIuserDataRegistered : public SEI32
+{33
+public:34
+ SEIuserDataRegistered()35
+ {36
+ m_payloadType = USER_DATA_REGISTERED_ITU_T_T35;37
+ m_payloadSize = 0;38
+ }39
+40
+ uint8_t *m_userData;41
+42
+ // daniel.vt@samsung.com :: for the Creative Intent Meta Data Encoding ( seongnam.oh@samsung.com )43
+ void writeSEI(const SPS&)44
+ {45
+ if (!m_userData)46
+ return;47
+48
+ uint32_t i = 0;49
+ for (; i < m_payloadSize; ++i)50
+ WRITE_CODE(m_userData[i], 8, "creative_intent_metadata");51
+ }52
+};53
+54
+static const uint32_t ISO_IEC_11578_LEN = 16;55
+56
class SEIuserDataUnregistered : public SEI57
{58
public:59
60
m_payloadType = USER_DATA_UNREGISTERED;61
m_payloadSize = 0;62
}63
- static const uint8_t m_uuid_iso_iec_11578[16];64
+ static const uint8_t m_uuid_iso_iec_11578[ISO_IEC_11578_LEN];65
uint8_t *m_userData;66
void writeSEI(const SPS&)67
{68
- for (uint32_t i = 0; i < 16; i++)69
+ for (uint32_t i = 0; i < ISO_IEC_11578_LEN; i++)70
WRITE_CODE(m_uuid_iso_iec_11578[i], 8, "sei.uuid_iso_iec_11578[i]");71
for (uint32_t i = 0; i < m_payloadSize; i++)72
WRITE_CODE(m_userData[i], 8, "user_data");73
74
CRC,75
CHECKSUM,76
} m_method;77
- uint8_t m_digest[3][16];78
+79
+ MD5Context m_state[3];80
+ uint32_t m_crc[3];81
+ uint32_t m_checksum[3];82
+ uint8_t m_digest[3][16];83
+84
void writeSEI(const SPS& sps)85
{86
int planes = (sps.chromaFormatIdc != X265_CSP_I400) ? 3 : 1;87
88
class SEIRecoveryPoint : public SEI89
{90
public:91
+ SEIRecoveryPoint()92
+ {93
+ m_payloadType = RECOVERY_POINT;94
+ m_payloadSize = 0;95
+ }96
int m_recoveryPocCnt;97
bool m_exactMatchingFlag;98
bool m_brokenLinkFlag;99
100
}101
};102
103
-//seongnam.oh@samsung.com :: for the Creative Intent Meta Data Encoding104
-class SEICreativeIntentMeta : public SEI105
+class SEIAlternativeTC : public SEI106
{107
public:108
- SEICreativeIntentMeta()109
+ int m_preferredTransferCharacteristics;110
+ SEIAlternativeTC()111
{112
- m_payloadType = USER_DATA_REGISTERED_ITU_T_T35;113
+ m_payloadType = ALTERNATIVE_TRANSFER_CHARACTERISTICS;114
m_payloadSize = 0;115
+ m_preferredTransferCharacteristics = -1;116
}117
118
- uint8_t *m_payload;119
-120
- // daniel.vt@samsung.com :: for the Creative Intent Meta Data Encoding ( seongnam.oh@samsung.com )121
void writeSEI(const SPS&)122
{123
- if (!m_payload)124
- return;125
-126
- uint32_t i = 0;127
- for (; i < m_payloadSize; ++i)128
- WRITE_CODE(m_payload[i], 8, "creative_intent_metadata");129
+ WRITE_CODE(m_preferredTransferCharacteristics, 8, "Preferred transfer characteristics");130
}131
};132
+133
}134
#endif // ifndef X265_SEI_H135
x265_2.7.tar.gz/source/encoder/slicetype.cpp -> x265_2.9.tar.gz/source/encoder/slicetype.cpp
Changed
400
1
2
curFrame->m_lowres.wp_sum[y] = 0;3
}4
5
- /* Calculate Qp offset for each 16x16 or 8x8 block in the frame */6
- int blockXY = 0;7
- int blockX = 0, blockY = 0;8
- double strength = 0.f;9
+ /* Calculate Qp offset for each 16x16 or 8x8 block in the frame */ 10
if ((param->rc.aqMode == X265_AQ_NONE || param->rc.aqStrength == 0) || (param->rc.bStatRead && param->rc.cuTree && IS_REFERENCED(curFrame)))11
{12
- /* Need to init it anyways for CU tree */13
- int cuCount = blockCount;14
-15
if (param->rc.aqMode && param->rc.aqStrength == 0)16
{17
if (quantOffsets)18
{19
- for (int cuxy = 0; cuxy < cuCount; cuxy++)20
+ for (int cuxy = 0; cuxy < blockCount; cuxy++)21
{22
curFrame->m_lowres.qpCuTreeOffset[cuxy] = curFrame->m_lowres.qpAqOffset[cuxy] = quantOffsets[cuxy];23
curFrame->m_lowres.invQscaleFactor[cuxy] = x265_exp2fix8(curFrame->m_lowres.qpCuTreeOffset[cuxy]);24
25
}26
else27
{28
- memset(curFrame->m_lowres.qpCuTreeOffset, 0, cuCount * sizeof(double));29
- memset(curFrame->m_lowres.qpAqOffset, 0, cuCount * sizeof(double));30
- for (int cuxy = 0; cuxy < cuCount; cuxy++)31
- curFrame->m_lowres.invQscaleFactor[cuxy] = 256;32
+ memset(curFrame->m_lowres.qpCuTreeOffset, 0, blockCount * sizeof(double));33
+ memset(curFrame->m_lowres.qpAqOffset, 0, blockCount * sizeof(double));34
+ for (int cuxy = 0; cuxy < blockCount; cuxy++)35
+ curFrame->m_lowres.invQscaleFactor[cuxy] = 256;36
}37
}38
39
- /* Need variance data for weighted prediction */40
+ /* Need variance data for weighted prediction and dynamic refinement*/41
if (param->bEnableWeightedPred || param->bEnableWeightedBiPred)42
{43
- for (blockY = 0; blockY < maxRow; blockY += loopIncr)44
- for (blockX = 0; blockX < maxCol; blockX += loopIncr)45
- acEnergyCu(curFrame, blockX, blockY, param->internalCsp, param->rc.qgSize);46
+ for (int blockY = 0; blockY < maxRow; blockY += loopIncr)47
+ for (int blockX = 0; blockX < maxCol; blockX += loopIncr) 48
+ acEnergyCu(curFrame, blockX, blockY, param->internalCsp, param->rc.qgSize); 49
}50
}51
else52
{53
- blockXY = 0;54
- double avg_adj_pow2 = 0, avg_adj = 0, qp_adj = 0;55
- double bias_strength = 0.f;56
+ int blockXY = 0;57
+ double avg_adj_pow2 = 0.f, avg_adj = 0.f, qp_adj = 0.f;58
+ double bias_strength = 0.f, strength = 0.f;59
if (param->rc.aqMode == X265_AQ_AUTO_VARIANCE || param->rc.aqMode == X265_AQ_AUTO_VARIANCE_BIASED)60
{61
- double bit_depth_correction = 1.f / (1 << (2*(X265_DEPTH-8)));62
- curFrame->m_lowres.frameVariance = 0;63
- uint64_t rowVariance = 0;64
- for (blockY = 0; blockY < maxRow; blockY += loopIncr)65
- {66
- rowVariance = 0;67
- for (blockX = 0; blockX < maxCol; blockX += loopIncr)68
- {69
- uint32_t energy = acEnergyCu(curFrame, blockX, blockY, param->internalCsp, param->rc.qgSize);70
- curFrame->m_lowres.blockVariance[blockXY] = energy;71
- rowVariance += energy;72
+ double bit_depth_correction = 1.f / (1 << (2*(X265_DEPTH-8))); 73
+ 74
+ for (int blockY = 0; blockY < maxRow; blockY += loopIncr)75
+ { 76
+ for (int blockX = 0; blockX < maxCol; blockX += loopIncr)77
+ {78
+ uint32_t energy = acEnergyCu(curFrame, blockX, blockY, param->internalCsp, param->rc.qgSize); 79
qp_adj = pow(energy * bit_depth_correction + 1, 0.1);80
curFrame->m_lowres.qpCuTreeOffset[blockXY] = qp_adj;81
avg_adj += qp_adj;82
avg_adj_pow2 += qp_adj * qp_adj;83
blockXY++;84
}85
- curFrame->m_lowres.frameVariance += (rowVariance / maxCol);86
}87
- curFrame->m_lowres.frameVariance /= maxRow;88
avg_adj /= blockCount;89
avg_adj_pow2 /= blockCount;90
strength = param->rc.aqStrength * avg_adj;91
- avg_adj = avg_adj - 0.5f * (avg_adj_pow2 - (modeTwoConst)) / avg_adj;92
+ avg_adj = avg_adj - 0.5f * (avg_adj_pow2 - modeTwoConst) / avg_adj;93
bias_strength = param->rc.aqStrength;94
}95
else96
strength = param->rc.aqStrength * 1.0397f;97
98
blockXY = 0;99
- for (blockY = 0; blockY < maxRow; blockY += loopIncr)100
+ for (int blockY = 0; blockY < maxRow; blockY += loopIncr)101
{102
- for (blockX = 0; blockX < maxCol; blockX += loopIncr)103
+ for (int blockX = 0; blockX < maxCol; blockX += loopIncr)104
{105
if (param->rc.aqMode == X265_AQ_AUTO_VARIANCE_BIASED)106
{107
108
else109
{110
uint32_t energy = acEnergyCu(curFrame, blockX, blockY, param->internalCsp,param->rc.qgSize);111
- qp_adj = strength * (X265_LOG2(X265_MAX(energy, 1)) - (modeOneConst + 2 * (X265_DEPTH - 8)));112
+ qp_adj = strength * (X265_LOG2(X265_MAX(energy, 1)) - (modeOneConst + 2 * (X265_DEPTH - 8))); 113
}114
115
if (param->bHDROpt)116
117
curFrame->m_lowres.wp_ssd[i] = ssd - (sum * sum + (width[i] * height[i]) / 2) / (width[i] * height[i]);118
}119
}120
+121
+ if (param->bDynamicRefine)122
+ {123
+ int blockXY = 0;124
+ for (int blockY = 0; blockY < maxRow; blockY += loopIncr)125
+ for (int blockX = 0; blockX < maxCol; blockX += loopIncr)126
+ {127
+ curFrame->m_lowres.blockVariance[blockXY] = acEnergyCu(curFrame, blockX, blockY, param->internalCsp, param->rc.qgSize);128
+ blockXY++;129
+ }130
+ }131
}132
133
void LookaheadTLD::lowresIntraEstimate(Lowres& fenc, uint32_t qgSize)134
135
pixel *src = ref.fpelPlane[0];136
intptr_t stride = fenc.lumaStride;137
138
- if (wp.bPresentFlag)139
+ if (wp.wtPresent)140
{141
int offset = wp.inputOffset << (X265_DEPTH - 8);142
int scale = wp.inputWeight;143
144
int deltaIndex = fenc.frameNum - ref.frameNum;145
146
WeightParam wp;147
- wp.bPresentFlag = false;148
+ wp.wtPresent = 0;149
150
if (!wbuffer[0])151
{152
153
}154
155
int bframes, brefs;156
- for (bframes = 0, brefs = 0;; bframes++)157
+ if (!m_param->analysisLoad)158
{159
- Lowres& frm = list[bframes]->m_lowres;160
-161
- if (frm.sliceType == X265_TYPE_BREF && !m_param->bBPyramid && brefs == m_param->bBPyramid)162
+ for (bframes = 0, brefs = 0;; bframes++)163
{164
- frm.sliceType = X265_TYPE_B;165
- x265_log(m_param, X265_LOG_WARNING, "B-ref at frame %d incompatible with B-pyramid\n",166
- frm.frameNum);167
- }168
+ Lowres& frm = list[bframes]->m_lowres;169
170
- /* pyramid with multiple B-refs needs a big enough dpb that the preceding P-frame stays available.171
- * smaller dpb could be supported by smart enough use of mmco, but it's easier just to forbid it. */172
- else if (frm.sliceType == X265_TYPE_BREF && m_param->bBPyramid && brefs &&173
- m_param->maxNumReferences <= (brefs + 3))174
- {175
- frm.sliceType = X265_TYPE_B;176
- x265_log(m_param, X265_LOG_WARNING, "B-ref at frame %d incompatible with B-pyramid and %d reference frames\n",177
- frm.sliceType, m_param->maxNumReferences);178
- }179
- if ((!m_param->bIntraRefresh || frm.frameNum == 0) && frm.frameNum - m_lastKeyframe >= m_param->keyframeMax &&180
- (!m_extendGopBoundary || frm.frameNum - m_lastKeyframe >= m_param->keyframeMax + m_param->gopLookahead))181
- {182
- if (frm.sliceType == X265_TYPE_AUTO || frm.sliceType == X265_TYPE_I)183
- frm.sliceType = m_param->bOpenGOP && m_lastKeyframe >= 0 ? X265_TYPE_I : X265_TYPE_IDR;184
- bool warn = frm.sliceType != X265_TYPE_IDR;185
- if (warn && m_param->bOpenGOP)186
- warn &= frm.sliceType != X265_TYPE_I;187
- if (warn)188
+ if (frm.sliceType == X265_TYPE_BREF && !m_param->bBPyramid && brefs == m_param->bBPyramid)189
{190
- x265_log(m_param, X265_LOG_WARNING, "specified frame type (%d) at %d is not compatible with keyframe interval\n",191
- frm.sliceType, frm.frameNum);192
- frm.sliceType = m_param->bOpenGOP && m_lastKeyframe >= 0 ? X265_TYPE_I : X265_TYPE_IDR;193
+ frm.sliceType = X265_TYPE_B;194
+ x265_log(m_param, X265_LOG_WARNING, "B-ref at frame %d incompatible with B-pyramid\n",195
+ frm.frameNum);196
}197
- }198
- if (frm.sliceType == X265_TYPE_I && frm.frameNum - m_lastKeyframe >= m_param->keyframeMin)199
- {200
- if (m_param->bOpenGOP)201
+202
+ /* pyramid with multiple B-refs needs a big enough dpb that the preceding P-frame stays available.203
+ * smaller dpb could be supported by smart enough use of mmco, but it's easier just to forbid it. */204
+ else if (frm.sliceType == X265_TYPE_BREF && m_param->bBPyramid && brefs &&205
+ m_param->maxNumReferences <= (brefs + 3))206
+ {207
+ frm.sliceType = X265_TYPE_B;208
+ x265_log(m_param, X265_LOG_WARNING, "B-ref at frame %d incompatible with B-pyramid and %d reference frames\n",209
+ frm.sliceType, m_param->maxNumReferences);210
+ }211
+ if (((!m_param->bIntraRefresh || frm.frameNum == 0) && frm.frameNum - m_lastKeyframe >= m_param->keyframeMax &&212
+ (!m_extendGopBoundary || frm.frameNum - m_lastKeyframe >= m_param->keyframeMax + m_param->gopLookahead)) ||213
+ (frm.frameNum == (m_param->chunkStart - 1)) || (frm.frameNum == m_param->chunkEnd))214
+ {215
+ if (frm.sliceType == X265_TYPE_AUTO || frm.sliceType == X265_TYPE_I)216
+ frm.sliceType = m_param->bOpenGOP && m_lastKeyframe >= 0 ? X265_TYPE_I : X265_TYPE_IDR;217
+ bool warn = frm.sliceType != X265_TYPE_IDR;218
+ if (warn && m_param->bOpenGOP)219
+ warn &= frm.sliceType != X265_TYPE_I;220
+ if (warn)221
+ {222
+ x265_log(m_param, X265_LOG_WARNING, "specified frame type (%d) at %d is not compatible with keyframe interval\n",223
+ frm.sliceType, frm.frameNum);224
+ frm.sliceType = m_param->bOpenGOP && m_lastKeyframe >= 0 ? X265_TYPE_I : X265_TYPE_IDR;225
+ }226
+ }227
+ if ((frm.sliceType == X265_TYPE_I && frm.frameNum - m_lastKeyframe >= m_param->keyframeMin) || (frm.frameNum == (m_param->chunkStart - 1)) || (frm.frameNum == m_param->chunkEnd))228
{229
+ if (m_param->bOpenGOP)230
+ {231
+ m_lastKeyframe = frm.frameNum;232
+ frm.bKeyframe = true;233
+ }234
+ else235
+ frm.sliceType = X265_TYPE_IDR;236
+ }237
+ if (frm.sliceType == X265_TYPE_IDR)238
+ {239
+ /* Closed GOP */240
m_lastKeyframe = frm.frameNum;241
frm.bKeyframe = true;242
+ if (bframes > 0 && !m_param->radl)243
+ {244
+ list[bframes - 1]->m_lowres.sliceType = X265_TYPE_P;245
+ bframes--;246
+ }247
}248
- else249
- frm.sliceType = X265_TYPE_IDR;250
- }251
- if (frm.sliceType == X265_TYPE_IDR)252
- {253
- /* Closed GOP */254
- m_lastKeyframe = frm.frameNum;255
- frm.bKeyframe = true;256
- if (bframes > 0 && !m_param->radl)257
+ if (bframes == m_param->bframes || !list[bframes + 1])258
{259
- list[bframes - 1]->m_lowres.sliceType = X265_TYPE_P;260
- bframes--;261
+ if (IS_X265_TYPE_B(frm.sliceType))262
+ x265_log(m_param, X265_LOG_WARNING, "specified frame type is not compatible with max B-frames\n");263
+ if (frm.sliceType == X265_TYPE_AUTO || IS_X265_TYPE_B(frm.sliceType))264
+ frm.sliceType = X265_TYPE_P;265
}266
- }267
- if (m_param->radl && !m_param->bOpenGOP && list[bframes + 1])268
- {269
- if ((frm.frameNum - m_lastKeyframe) > (m_param->keyframeMax - m_param->radl - 1) && (frm.frameNum - m_lastKeyframe) < m_param->keyframeMax)270
+ if (frm.sliceType == X265_TYPE_BREF)271
+ brefs++;272
+ if (frm.sliceType == X265_TYPE_AUTO)273
frm.sliceType = X265_TYPE_B;274
- if ((frm.frameNum - m_lastKeyframe) == (m_param->keyframeMax - m_param->radl - 1))275
- frm.sliceType = X265_TYPE_P;276
+ else if (!IS_X265_TYPE_B(frm.sliceType))277
+ break;278
}279
-280
- if (bframes == m_param->bframes || !list[bframes + 1])281
+ }282
+ else283
+ {284
+ for (bframes = 0, brefs = 0;; bframes++)285
{286
- if (IS_X265_TYPE_B(frm.sliceType))287
- x265_log(m_param, X265_LOG_WARNING, "specified frame type is not compatible with max B-frames\n");288
- if (frm.sliceType == X265_TYPE_AUTO || IS_X265_TYPE_B(frm.sliceType))289
- frm.sliceType = X265_TYPE_P;290
- }291
- if (frm.sliceType == X265_TYPE_BREF)292
- brefs++;293
- if (frm.sliceType == X265_TYPE_AUTO)294
- frm.sliceType = X265_TYPE_B;295
- else if (!IS_X265_TYPE_B(frm.sliceType))296
- break;297
+ Lowres& frm = list[bframes]->m_lowres;298
+ if (frm.sliceType == X265_TYPE_BREF)299
+ brefs++;300
+ if ((IS_X265_TYPE_I(frm.sliceType) && frm.frameNum - m_lastKeyframe >= m_param->keyframeMin)301
+ || (frm.frameNum == (m_param->chunkStart - 1)) || (frm.frameNum == m_param->chunkEnd))302
+ {303
+ m_lastKeyframe = frm.frameNum;304
+ frm.bKeyframe = true;305
+ }306
+ if (!IS_X265_TYPE_B(frm.sliceType))307
+ break;308
+ }309
}310
-311
if (bframes)312
list[bframes - 1]->m_lowres.bLastMiniGopBFrame = true;313
list[bframes]->m_lowres.leadingBframes = bframes;314
315
return;316
}317
frames[framecnt + 1] = NULL;318
- int keyFrameLimit = m_param->keyframeMax + m_lastKeyframe - frames[0]->frameNum - 1;319
+320
+ int keylimit = m_param->keyframeMax;321
+ if (frames[0]->frameNum < m_param->chunkEnd)322
+ {323
+ int chunkStart = (m_param->chunkStart - m_lastKeyframe - 1);324
+ int chunkEnd = (m_param->chunkEnd - m_lastKeyframe);325
+ if ((chunkStart > 0) && (chunkStart < m_param->keyframeMax))326
+ keylimit = chunkStart;327
+ else if ((chunkEnd > 0) && (chunkEnd < m_param->keyframeMax))328
+ keylimit = chunkEnd;329
+ }330
+331
+ int keyFrameLimit = keylimit + m_lastKeyframe - frames[0]->frameNum - 1;332
if (m_param->gopLookahead && keyFrameLimit <= m_param->bframes + 1)333
keyintLimit = keyFrameLimit + m_param->gopLookahead;334
else335
336
int numBFrames = 0;337
int numAnalyzed = numFrames;338
bool isScenecut = scenecut(frames, 0, 1, true, origNumFrames);339
+340
/* When scenecut threshold is set, use scenecut detection for I frame placements */341
if (m_param->scenecutThreshold && isScenecut)342
{343
344
frames[numFrames]->sliceType = X265_TYPE_P;345
}346
347
- /* Check scenecut on the first minigop. */348
- for (int j = 1; j < numBFrames + 1; j++)349
+ bool bForceRADL = m_param->radl && !m_param->bOpenGOP;350
+ bool bLastMiniGop = (framecnt >= m_param->bframes + 1) ? false : true;351
+ int preRADL = m_lastKeyframe + m_param->keyframeMax - m_param->radl - 1; /*Frame preceeding RADL in POC order*/352
+ if (bForceRADL && (frames[0]->frameNum == preRADL) && !bLastMiniGop)353
+ {354
+ int j = 1;355
+ numBFrames = m_param->radl;356
+ for (; j <= m_param->radl; j++)357
+ frames[j]->sliceType = X265_TYPE_B;358
+ frames[j]->sliceType = X265_TYPE_I;359
+ }360
+ else /* Check scenecut and RADL on the first minigop. */361
{362
- if (scenecut(frames, j, j + 1, false, origNumFrames))363
+ for (int j = 1; j < numBFrames + 1; j++)364
{365
- frames[j]->sliceType = X265_TYPE_P;366
- numAnalyzed = j;367
- break;368
+ if (scenecut(frames, j, j + 1, false, origNumFrames) || 369
+ (bForceRADL && (frames[j]->frameNum == preRADL)))370
+ {371
+ frames[j]->sliceType = X265_TYPE_P;372
+ numAnalyzed = j;373
+ break;374
+ }375
}376
}377
resetStart = bKeyframe ? 1 : X265_MIN(numBFrames + 2, numAnalyzed + 1);378
379
intptr_t stride0 = X265_LOWRES_CU_SIZE, stride1 = X265_LOWRES_CU_SIZE;380
pixel *src0 = fref0->lowresMC(pelOffset, fenc->lowresMvs[0][listDist[0]][cuXY], subpelbuf0, stride0);381
pixel *src1 = fref1->lowresMC(pelOffset, fenc->lowresMvs[1][listDist[1]][cuXY], subpelbuf1, stride1);382
-383
ALIGN_VAR_32(pixel, ref[X265_LOWRES_CU_SIZE * X265_LOWRES_CU_SIZE]);384
- primitives.pu[LUMA_8x8].pixelavg_pp(ref, X265_LOWRES_CU_SIZE, src0, stride0, src1, stride1, 32);385
+ primitives.pu[LUMA_8x8].pixelavg_pp[NONALIGNED](ref, X265_LOWRES_CU_SIZE, src0, stride0, src1, stride1, 32);386
int bicost = tld.me.bufSATD(ref, X265_LOWRES_CU_SIZE);387
COPY2_IF_LT(bcost, bicost, listused, 3);388
-389
/* coloc candidate */390
src0 = fref0->lowresPlane[0] + pelOffset;391
src1 = fref1->lowresPlane[0] + pelOffset;392
- primitives.pu[LUMA_8x8].pixelavg_pp(ref, X265_LOWRES_CU_SIZE, src0, fref0->lumaStride, src1, fref1->lumaStride, 32);393
+ primitives.pu[LUMA_8x8].pixelavg_pp[NONALIGNED](ref, X265_LOWRES_CU_SIZE, src0, fref0->lumaStride, src1, fref1->lumaStride, 32);394
bicost = tld.me.bufSATD(ref, X265_LOWRES_CU_SIZE);395
COPY2_IF_LT(bcost, bicost, listused, 3);396
-397
bcost += lowresPenalty;398
}399
else /* P, also consider intra */400
x265_2.7.tar.gz/source/encoder/weightPrediction.cpp -> x265_2.9.tar.gz/source/encoder/weightPrediction.cpp
Changed
56
1
2
int denom = w->log2WeightDenom;3
int round = denom ? 1 << (denom - 1) : 0;4
int correction = IF_INTERNAL_PREC - X265_DEPTH; /* intermediate interpolation depth */5
- int pwidth = ((width + 15) >> 4) << 4;6
-7
+ int pwidth = ((width + 31) >> 5) << 5;8
primitives.weight_pp(ref, weightTemp, stride, pwidth, height,9
weight, round << correction, denom + correction, offset);10
ref = weightTemp;11
12
for (int plane = 0; plane < (param.internalCsp != X265_CSP_I400 ? 3 : 1); plane++)13
{14
denom = plane ? chromaDenom : lumaDenom;15
- if (plane && !weights[0].bPresentFlag)16
+ if (plane && !weights[0].wtPresent)17
break;18
19
/* Early termination */20
21
}22
}23
24
- if (weights[0].bPresentFlag)25
+ if (weights[0].wtPresent)26
{27
// Make sure both chroma channels match28
- if (weights[1].bPresentFlag != weights[2].bPresentFlag)29
+ if (weights[1].wtPresent != weights[2].wtPresent)30
{31
- if (weights[1].bPresentFlag)32
+ if (weights[1].wtPresent)33
weights[2] = weights[1];34
else35
weights[1] = weights[2];36
37
for (int list = 0; list < numPredDir; list++)38
{39
WeightParam* w = &wp[list][0][0];40
- if (w[0].bPresentFlag || w[1].bPresentFlag || w[2].bPresentFlag)41
+ if (w[0].wtPresent || w[1].wtPresent || w[2].wtPresent)42
{43
bWeighted = true;44
p += sprintf(buf + p, " [L%d:R0 ", list);45
- if (w[0].bPresentFlag)46
+ if (w[0].wtPresent)47
p += sprintf(buf + p, "Y{%d/%d%+d}", w[0].inputWeight, 1 << w[0].log2WeightDenom, w[0].inputOffset);48
- if (w[1].bPresentFlag)49
+ if (w[1].wtPresent)50
p += sprintf(buf + p, "U{%d/%d%+d}", w[1].inputWeight, 1 << w[1].log2WeightDenom, w[1].inputOffset);51
- if (w[2].bPresentFlag)52
+ if (w[2].wtPresent)53
p += sprintf(buf + p, "V{%d/%d%+d}", w[2].inputWeight, 1 << w[2].log2WeightDenom, w[2].inputOffset);54
p += sprintf(buf + p, "]");55
}56
x265_2.7.tar.gz/source/test/ipfilterharness.cpp -> x265_2.9.tar.gz/source/test/ipfilterharness.cpp
Changed
268
1
2
return true;3
}4
5
+bool IPFilterHarness::check_IPFilterLumaP2S_aligned_primitive(filter_p2s_t ref, filter_p2s_t opt)6
+{7
+ for (int i = 0; i < TEST_CASES; i++)8
+ {9
+ int index = i % TEST_CASES;10
+ intptr_t rand_srcStride[] = { 128, 192, 256, 512 };11
+ intptr_t dstStride[] = { 192, 256, 512, 576 };12
+ for (int p = 0; p < 4; p++)13
+ {14
+ ref(pixel_test_buff[index], rand_srcStride[p], IPF_C_output_s, dstStride[p]);15
+ checked(opt, pixel_test_buff[index] + (64 * i), rand_srcStride[p], IPF_vec_output_s, dstStride[p]);16
+ if (memcmp(IPF_vec_output_s, IPF_C_output_s, TEST_BUF_SIZE * sizeof(int16_t)))17
+ return false;18
+ }19
+ reportfail();20
+ }21
+22
+ return true;23
+}24
+25
bool IPFilterHarness::check_IPFilterChromaP2S_primitive(filter_p2s_t ref, filter_p2s_t opt)26
{27
for (int i = 0; i < ITERS; i++)28
29
return true;30
}31
32
+bool IPFilterHarness::check_IPFilterChromaP2S_aligned_primitive(filter_p2s_t ref, filter_p2s_t opt)33
+{34
+ for (int i = 0; i < TEST_CASES; i++)35
+ {36
+ int index = i % TEST_CASES;37
+ intptr_t rand_srcStride[] = { 128, 192, 256, 512};38
+ intptr_t dstStride[] = { 192, 256, 512, 576 };39
+40
+ for (int p = 0; p < 4; p++)41
+ {42
+ ref(pixel_test_buff[index], rand_srcStride[p], IPF_C_output_s, dstStride[p]);43
+44
+ checked(opt, pixel_test_buff[index], rand_srcStride[p], IPF_vec_output_s, dstStride[p]);45
+46
+ if (memcmp(IPF_vec_output_s, IPF_C_output_s, TEST_BUF_SIZE * sizeof(int16_t)))47
+ return false;48
+ }49
+ reportfail();50
+ }51
+52
+ return true;53
+}54
+55
bool IPFilterHarness::testCorrectness(const EncoderPrimitives& ref, const EncoderPrimitives& opt)56
{57
58
59
return false;60
}61
}62
- if (opt.pu[value].convert_p2s)63
+ if (opt.pu[value].convert_p2s[NONALIGNED])64
{65
- if (!check_IPFilterLumaP2S_primitive(ref.pu[value].convert_p2s, opt.pu[value].convert_p2s))66
+ if (!check_IPFilterLumaP2S_primitive(ref.pu[value].convert_p2s[NONALIGNED], opt.pu[value].convert_p2s[NONALIGNED]))67
{68
printf("convert_p2s[%s]", lumaPartStr[value]);69
return false;70
}71
}72
+ if (opt.pu[value].convert_p2s[ALIGNED])73
+ {74
+ if (!check_IPFilterLumaP2S_aligned_primitive(ref.pu[value].convert_p2s[ALIGNED], opt.pu[value].convert_p2s[ALIGNED]))75
+ {76
+ printf("convert_p2s_aligned[%s]", lumaPartStr[value]);77
+ return false;78
+ }79
+ }80
}81
82
for (int csp = X265_CSP_I420; csp < X265_CSP_COUNT; csp++)83
84
return false;85
}86
}87
- if (opt.chroma[csp].pu[value].p2s)88
+ if (opt.chroma[csp].pu[value].p2s[ALIGNED])89
+ {90
+ if (!check_IPFilterChromaP2S_aligned_primitive(ref.chroma[csp].pu[value].p2s[ALIGNED], opt.chroma[csp].pu[value].p2s[ALIGNED]))91
+ {92
+ printf("chroma_p2s_aligned[%s]", chromaPartStr[csp][value]);93
+ return false;94
+ }95
+ }96
+ if (opt.chroma[csp].pu[value].p2s[NONALIGNED])97
{98
- if (!check_IPFilterChromaP2S_primitive(ref.chroma[csp].pu[value].p2s, opt.chroma[csp].pu[value].p2s))99
+ if (!check_IPFilterChromaP2S_primitive(ref.chroma[csp].pu[value].p2s[NONALIGNED], opt.chroma[csp].pu[value].p2s[NONALIGNED]))100
{101
printf("chroma_p2s[%s]", chromaPartStr[csp][value]);102
return false;103
104
105
void IPFilterHarness::measureSpeed(const EncoderPrimitives& ref, const EncoderPrimitives& opt)106
{107
- int16_t srcStride = 96;108
- int16_t dstStride = 96;109
+ int16_t srcStride = 192; /* Multiple of 64 */110
+ int16_t dstStride = 192;111
int maxVerticalfilterHalfDistance = 3;112
113
for (int value = 0; value < NUM_PU_SIZES; value++)114
115
{116
printf("luma_hpp[%s]\t", lumaPartStr[value]);117
REPORT_SPEEDUP(opt.pu[value].luma_hpp, ref.pu[value].luma_hpp,118
- pixel_buff + srcStride, srcStride, IPF_vec_output_p, dstStride, 1);119
+ pixel_buff + srcStride, srcStride, IPF_vec_output_p, dstStride, 1);120
}121
122
if (opt.pu[value].luma_hps)123
{124
printf("luma_hps[%s]\t", lumaPartStr[value]);125
REPORT_SPEEDUP(opt.pu[value].luma_hps, ref.pu[value].luma_hps,126
- pixel_buff + maxVerticalfilterHalfDistance * srcStride, srcStride,127
- IPF_vec_output_s, dstStride, 1, 1);128
+ pixel_buff + maxVerticalfilterHalfDistance * srcStride, srcStride,129
+ IPF_vec_output_s, dstStride, 1, 1);130
}131
132
if (opt.pu[value].luma_vpp)133
{134
printf("luma_vpp[%s]\t", lumaPartStr[value]);135
REPORT_SPEEDUP(opt.pu[value].luma_vpp, ref.pu[value].luma_vpp,136
- pixel_buff + maxVerticalfilterHalfDistance * srcStride, srcStride,137
- IPF_vec_output_p, dstStride, 1);138
+ pixel_buff + maxVerticalfilterHalfDistance * srcStride, srcStride,139
+ IPF_vec_output_p, dstStride, 1);140
}141
142
if (opt.pu[value].luma_vps)143
{144
printf("luma_vps[%s]\t", lumaPartStr[value]);145
REPORT_SPEEDUP(opt.pu[value].luma_vps, ref.pu[value].luma_vps,146
- pixel_buff + maxVerticalfilterHalfDistance * srcStride, srcStride,147
- IPF_vec_output_s, dstStride, 1);148
+ pixel_buff + maxVerticalfilterHalfDistance * srcStride, srcStride,149
+ IPF_vec_output_s, dstStride, 1);150
}151
152
if (opt.pu[value].luma_vsp)153
{154
printf("luma_vsp[%s]\t", lumaPartStr[value]);155
REPORT_SPEEDUP(opt.pu[value].luma_vsp, ref.pu[value].luma_vsp,156
- short_buff + maxVerticalfilterHalfDistance * srcStride, srcStride,157
- IPF_vec_output_p, dstStride, 1);158
+ short_buff + maxVerticalfilterHalfDistance * srcStride, srcStride,159
+ IPF_vec_output_p, dstStride, 1);160
}161
162
if (opt.pu[value].luma_vss)163
{164
printf("luma_vss[%s]\t", lumaPartStr[value]);165
REPORT_SPEEDUP(opt.pu[value].luma_vss, ref.pu[value].luma_vss,166
- short_buff + maxVerticalfilterHalfDistance * srcStride, srcStride,167
- IPF_vec_output_s, dstStride, 1);168
+ short_buff + maxVerticalfilterHalfDistance * srcStride, srcStride,169
+ IPF_vec_output_s, dstStride, 1);170
}171
172
if (opt.pu[value].luma_hvpp)173
{174
printf("luma_hv [%s]\t", lumaPartStr[value]);175
REPORT_SPEEDUP(opt.pu[value].luma_hvpp, ref.pu[value].luma_hvpp,176
- pixel_buff + 3 * srcStride, srcStride, IPF_vec_output_p, srcStride, 1, 3);177
+ pixel_buff + 3 * srcStride, srcStride, IPF_vec_output_p, srcStride, 1, 3);178
}179
180
- if (opt.pu[value].convert_p2s)181
+ if (opt.pu[value].convert_p2s[NONALIGNED])182
{183
printf("convert_p2s[%s]\t", lumaPartStr[value]);184
- REPORT_SPEEDUP(opt.pu[value].convert_p2s, ref.pu[value].convert_p2s,185
- pixel_buff, srcStride,186
- IPF_vec_output_s, dstStride);187
+ REPORT_SPEEDUP(opt.pu[value].convert_p2s[NONALIGNED], ref.pu[value].convert_p2s[NONALIGNED],188
+ pixel_buff, srcStride,189
+ IPF_vec_output_s, dstStride);190
+ }191
+192
+ if (opt.pu[value].convert_p2s[ALIGNED])193
+ {194
+ printf("convert_p2s_aligned[%s]\t", lumaPartStr[value]);195
+ REPORT_SPEEDUP(opt.pu[value].convert_p2s[ALIGNED], ref.pu[value].convert_p2s[ALIGNED],196
+ pixel_buff, srcStride,197
+ IPF_vec_output_s, dstStride);198
}199
}200
201
202
{203
printf("chroma_hpp[%s]", chromaPartStr[csp][value]);204
REPORT_SPEEDUP(opt.chroma[csp].pu[value].filter_hpp, ref.chroma[csp].pu[value].filter_hpp,205
- pixel_buff + srcStride, srcStride, IPF_vec_output_p, dstStride, 1);206
+ pixel_buff + srcStride, srcStride, IPF_vec_output_p, dstStride, 1);207
}208
if (opt.chroma[csp].pu[value].filter_hps)209
{210
printf("chroma_hps[%s]", chromaPartStr[csp][value]);211
REPORT_SPEEDUP(opt.chroma[csp].pu[value].filter_hps, ref.chroma[csp].pu[value].filter_hps,212
- pixel_buff + srcStride, srcStride, IPF_vec_output_s, dstStride, 1, 1);213
+ pixel_buff + srcStride, srcStride, IPF_vec_output_s, dstStride, 1, 1);214
}215
if (opt.chroma[csp].pu[value].filter_vpp)216
{217
printf("chroma_vpp[%s]", chromaPartStr[csp][value]);218
REPORT_SPEEDUP(opt.chroma[csp].pu[value].filter_vpp, ref.chroma[csp].pu[value].filter_vpp,219
- pixel_buff + maxVerticalfilterHalfDistance * srcStride, srcStride,220
- IPF_vec_output_p, dstStride, 1);221
+ pixel_buff + maxVerticalfilterHalfDistance * srcStride, srcStride,222
+ IPF_vec_output_p, dstStride, 1);223
}224
if (opt.chroma[csp].pu[value].filter_vps)225
{226
printf("chroma_vps[%s]", chromaPartStr[csp][value]);227
REPORT_SPEEDUP(opt.chroma[csp].pu[value].filter_vps, ref.chroma[csp].pu[value].filter_vps,228
- pixel_buff + maxVerticalfilterHalfDistance * srcStride, srcStride,229
- IPF_vec_output_s, dstStride, 1);230
+ pixel_buff + maxVerticalfilterHalfDistance * srcStride, srcStride,231
+ IPF_vec_output_s, dstStride, 1);232
}233
if (opt.chroma[csp].pu[value].filter_vsp)234
{235
printf("chroma_vsp[%s]", chromaPartStr[csp][value]);236
REPORT_SPEEDUP(opt.chroma[csp].pu[value].filter_vsp, ref.chroma[csp].pu[value].filter_vsp,237
- short_buff + maxVerticalfilterHalfDistance * srcStride, srcStride,238
- IPF_vec_output_p, dstStride, 1);239
+ short_buff + maxVerticalfilterHalfDistance * srcStride, srcStride,240
+ IPF_vec_output_p, dstStride, 1);241
}242
if (opt.chroma[csp].pu[value].filter_vss)243
{244
printf("chroma_vss[%s]", chromaPartStr[csp][value]);245
REPORT_SPEEDUP(opt.chroma[csp].pu[value].filter_vss, ref.chroma[csp].pu[value].filter_vss,246
- short_buff + maxVerticalfilterHalfDistance * srcStride, srcStride,247
- IPF_vec_output_s, dstStride, 1);248
+ short_buff + maxVerticalfilterHalfDistance * srcStride, srcStride,249
+ IPF_vec_output_s, dstStride, 1);250
}251
- if (opt.chroma[csp].pu[value].p2s)252
+ if (opt.chroma[csp].pu[value].p2s[NONALIGNED])253
{254
printf("chroma_p2s[%s]\t", chromaPartStr[csp][value]);255
- REPORT_SPEEDUP(opt.chroma[csp].pu[value].p2s, ref.chroma[csp].pu[value].p2s,256
- pixel_buff, srcStride, IPF_vec_output_s, dstStride);257
+ REPORT_SPEEDUP(opt.chroma[csp].pu[value].p2s[NONALIGNED], ref.chroma[csp].pu[value].p2s[NONALIGNED],258
+ pixel_buff, srcStride, IPF_vec_output_s, dstStride);259
+ }260
+ if (opt.chroma[csp].pu[value].p2s[ALIGNED])261
+ {262
+ printf("chroma_p2s_aligned[%s]\t", chromaPartStr[csp][value]);263
+ REPORT_SPEEDUP(opt.chroma[csp].pu[value].p2s[ALIGNED], ref.chroma[csp].pu[value].p2s[ALIGNED],264
+ pixel_buff, srcStride, IPF_vec_output_s, dstStride);265
}266
}267
}268
x265_2.7.tar.gz/source/test/ipfilterharness.h -> x265_2.9.tar.gz/source/test/ipfilterharness.h
Changed
35
1
2
enum { TEST_CASES = 3 };3
enum { SMAX = 1 << 12 };4
enum { SMIN = (unsigned)-1 << 12 };5
- ALIGN_VAR_32(pixel, pixel_buff[TEST_BUF_SIZE]);6
- int16_t short_buff[TEST_BUF_SIZE];7
- int16_t IPF_vec_output_s[TEST_BUF_SIZE];8
- int16_t IPF_C_output_s[TEST_BUF_SIZE];9
- pixel IPF_vec_output_p[TEST_BUF_SIZE];10
- pixel IPF_C_output_p[TEST_BUF_SIZE];11
+ ALIGN_VAR_64(pixel, pixel_buff[TEST_BUF_SIZE]);12
+ ALIGN_VAR_64(int16_t, short_buff[TEST_BUF_SIZE]);13
+ ALIGN_VAR_64(int16_t, IPF_vec_output_s[TEST_BUF_SIZE]);14
+ ALIGN_VAR_64(int16_t, IPF_C_output_s[TEST_BUF_SIZE]);15
+ ALIGN_VAR_64(pixel, IPF_vec_output_p[TEST_BUF_SIZE]);16
+ ALIGN_VAR_64(pixel, IPF_C_output_p[TEST_BUF_SIZE]);17
18
- pixel pixel_test_buff[TEST_CASES][TEST_BUF_SIZE];19
- int16_t short_test_buff[TEST_CASES][TEST_BUF_SIZE];20
+ ALIGN_VAR_64(pixel, pixel_test_buff[TEST_CASES][TEST_BUF_SIZE]);21
+ ALIGN_VAR_64(int16_t, short_test_buff[TEST_CASES][TEST_BUF_SIZE]);22
23
bool check_IPFilterChroma_primitive(filter_pp_t ref, filter_pp_t opt);24
bool check_IPFilterChroma_ps_primitive(filter_ps_t ref, filter_ps_t opt);25
26
bool check_IPFilterLuma_ss_primitive(filter_ss_t ref, filter_ss_t opt);27
bool check_IPFilterLumaHV_primitive(filter_hv_pp_t ref, filter_hv_pp_t opt);28
bool check_IPFilterLumaP2S_primitive(filter_p2s_t ref, filter_p2s_t opt);29
+ bool check_IPFilterLumaP2S_aligned_primitive(filter_p2s_t ref, filter_p2s_t opt);30
bool check_IPFilterChromaP2S_primitive(filter_p2s_t ref, filter_p2s_t opt);31
+ bool check_IPFilterChromaP2S_aligned_primitive(filter_p2s_t ref, filter_p2s_t opt);32
33
public:34
35
x265_2.7.tar.gz/source/test/mbdstharness.cpp -> x265_2.9.tar.gz/source/test/mbdstharness.cpp
Changed
256
1
2
for (int i = 0; i < TEST_BUF_SIZE; i++)3
{4
short_test_buff[0][i] = (rand() & PIXEL_MAX) - (rand() & PIXEL_MAX);5
+ short_test_buff1[0][i] = (rand() & PIXEL_MAX) - (rand() & PIXEL_MAX);6
int_test_buff[0][i] = rand() % PIXEL_MAX;7
int_idct_test_buff[0][i] = (rand() % (SHORT_MAX - SHORT_MIN)) - SHORT_MAX;8
short_denoise_test_buff1[0][i] = short_denoise_test_buff2[0][i] = (rand() & SHORT_MAX) - (rand() & SHORT_MAX);9
-10
short_test_buff[1][i] = -PIXEL_MAX;11
+ short_test_buff1[1][i] = -PIXEL_MAX;12
int_test_buff[1][i] = -PIXEL_MAX;13
int_idct_test_buff[1][i] = SHORT_MIN;14
short_denoise_test_buff1[1][i] = short_denoise_test_buff2[1][i] = -SHORT_MAX;15
-16
short_test_buff[2][i] = PIXEL_MAX;17
+ short_test_buff1[2][i] = PIXEL_MAX;18
int_test_buff[2][i] = PIXEL_MAX;19
int_idct_test_buff[2][i] = SHORT_MAX;20
short_denoise_test_buff1[2][i] = short_denoise_test_buff2[2][i] = SHORT_MAX;21
22
bool MBDstHarness::check_nquant_primitive(nquant_t ref, nquant_t opt)23
{24
int j = 0;25
-26
for (int i = 0; i < ITERS; i++)27
{28
- int width = (rand() % 4 + 1) * 4;29
+ int width = 1 << (rand() % 4 + 2);30
int height = width;31
-32
uint32_t optReturnValue = 0;33
uint32_t refReturnValue = 0;34
35
36
reportfail();37
j += INCR;38
}39
+ return true;40
+}41
+42
+bool MBDstHarness::check_nonPsyRdoQuant_primitive(nonPsyRdoQuant_t ref, nonPsyRdoQuant_t opt)43
+{44
+ int j = 0;45
+ int trSize[4] = { 16, 64, 256, 1024 };46
+47
+ ALIGN_VAR_32(int64_t, ref_dest[4 * MAX_TU_SIZE]);48
+ ALIGN_VAR_32(int64_t, opt_dest[4 * MAX_TU_SIZE]);49
+50
+ for (int i = 0; i < ITERS; i++)51
+ {52
+ int64_t totalRdCostRef = rand();53
+ int64_t totalUncodedCostRef = rand();54
+ int64_t totalRdCostOpt = totalRdCostRef;55
+ int64_t totalUncodedCostOpt = totalUncodedCostRef;56
+57
+ int index = rand() % 4;58
+ uint32_t blkPos = trSize[index];59
+ int cmp_size = 4 * MAX_TU_SIZE;60
+61
+ memset(ref_dest, 0, MAX_TU_SIZE * sizeof(int64_t));62
+ memset(opt_dest, 0, MAX_TU_SIZE * sizeof(int64_t));63
+64
+ int index1 = rand() % TEST_CASES;65
+66
+ ref(short_test_buff[index1] + j, ref_dest, &totalUncodedCostRef, &totalRdCostRef, blkPos);67
+ checked(opt, short_test_buff[index1] + j, opt_dest, &totalUncodedCostOpt, &totalRdCostOpt, blkPos);68
+69
+ if (memcmp(ref_dest, opt_dest, cmp_size))70
+ return false;71
+72
+ if (totalUncodedCostRef != totalUncodedCostOpt)73
+ return false;74
+75
+ if (totalRdCostRef != totalRdCostOpt)76
+ return false;77
+78
+ reportfail();79
+ j += INCR;80
+ }81
+82
+ return true;83
+}84
+bool MBDstHarness::check_psyRdoQuant_primitive(psyRdoQuant_t ref, psyRdoQuant_t opt)85
+{86
+ int j = 0;87
+ int trSize[4] = { 16, 64, 256, 1024 };88
+89
+ ALIGN_VAR_32(int64_t, ref_dest[4 * MAX_TU_SIZE]);90
+ ALIGN_VAR_32(int64_t, opt_dest[4 * MAX_TU_SIZE]);91
+92
+ for (int i = 0; i < ITERS; i++)93
+ {94
+ int64_t totalRdCostRef = rand();95
+ int64_t totalUncodedCostRef = rand();96
+ int64_t totalRdCostOpt = totalRdCostRef;97
+ int64_t totalUncodedCostOpt = totalUncodedCostRef;98
+ int64_t *psyScale = X265_MALLOC(int64_t, 1);99
+ *psyScale = rand();100
+101
+ int index = rand() % 4;102
+ uint32_t blkPos = trSize[index];103
+ int cmp_size = 4 * MAX_TU_SIZE;104
+105
+ memset(ref_dest, 0, MAX_TU_SIZE * sizeof(int64_t));106
+ memset(opt_dest, 0, MAX_TU_SIZE * sizeof(int64_t));107
+108
+ int index1 = rand() % TEST_CASES;109
+110
+ ref(short_test_buff[index1] + j, short_test_buff1[index1] + j, ref_dest, &totalUncodedCostRef, &totalRdCostRef, psyScale, blkPos);111
+ checked(opt, short_test_buff[index1] + j, short_test_buff1[index1] + j, opt_dest, &totalUncodedCostOpt, &totalRdCostOpt, psyScale, blkPos);112
+113
+ X265_FREE(psyScale);114
+ if (memcmp(ref_dest, opt_dest, cmp_size))115
+ return false;116
+117
+ if (totalUncodedCostRef != totalUncodedCostOpt)118
+ return false;119
+120
+ if (totalRdCostRef != totalRdCostOpt)121
+ return false;122
+123
+ reportfail();124
+ j += INCR;125
+ }126
+127
+ return true;128
+}129
+bool MBDstHarness::check_psyRdoQuant_primitive_avx2(psyRdoQuant_t1 ref, psyRdoQuant_t1 opt)130
+{131
+ int j = 0;132
+ int trSize[4] = { 16, 64, 256, 1024 };133
+134
+ ALIGN_VAR_32(int64_t, ref_dest[4 * MAX_TU_SIZE]);135
+ ALIGN_VAR_32(int64_t, opt_dest[4 * MAX_TU_SIZE]);136
+137
+ for (int i = 0; i < ITERS; i++)138
+ {139
+ int64_t totalRdCostRef = rand();140
+ int64_t totalUncodedCostRef = rand();141
+ int64_t totalRdCostOpt = totalRdCostRef;142
+ int64_t totalUncodedCostOpt = totalUncodedCostRef;143
+144
+ int index = rand() % 4;145
+ uint32_t blkPos = trSize[index];146
+ int cmp_size = 4 * MAX_TU_SIZE;147
+148
+ memset(ref_dest, 0, MAX_TU_SIZE * sizeof(int64_t));149
+ memset(opt_dest, 0, MAX_TU_SIZE * sizeof(int64_t));150
+151
+ int index1 = rand() % TEST_CASES;152
+153
+ ref(short_test_buff[index1] + j, ref_dest, &totalUncodedCostRef, &totalRdCostRef, blkPos);154
+ checked(opt, short_test_buff[index1] + j, opt_dest, &totalUncodedCostOpt, &totalRdCostOpt, blkPos);155
+156
+ 157
+ if (memcmp(ref_dest, opt_dest, cmp_size))158
+ return false;159
+160
+ if (totalUncodedCostRef != totalUncodedCostOpt)161
+ return false;162
+163
+ if (totalRdCostRef != totalRdCostOpt)164
+ return false;165
+166
+ reportfail();167
+ j += INCR;168
+ }169
170
return true;171
}172
173
return false;174
}175
}176
+177
+ for (int i = 0; i < NUM_TR_SIZE; i++)178
+ {179
+ if (opt.cu[i].nonPsyRdoQuant)180
+ {181
+ if (!check_nonPsyRdoQuant_primitive(ref.cu[i].nonPsyRdoQuant, opt.cu[i].nonPsyRdoQuant))182
+ {183
+ printf("nonPsyRdoQuant[%dx%d]: Failed!\n", 4 << i, 4 << i);184
+ return false;185
+ }186
+ }187
+ }188
+ for (int i = 0; i < NUM_TR_SIZE; i++)189
+ {190
+ if (opt.cu[i].psyRdoQuant)191
+ {192
+ if (!check_psyRdoQuant_primitive(ref.cu[i].psyRdoQuant, opt.cu[i].psyRdoQuant))193
+ {194
+ printf("psyRdoQuant[%dx%d]: Failed!\n", 4 << i, 4 << i);195
+ return false;196
+ }197
+ }198
+ }199
+ for (int i = 0; i < NUM_TR_SIZE; i++)200
+ {201
+ if (opt.cu[i].psyRdoQuant_1p)202
+ {203
+ if (!check_psyRdoQuant_primitive_avx2(ref.cu[i].psyRdoQuant_1p, opt.cu[i].psyRdoQuant_1p))204
+ {205
+ printf("psyRdoQuant_1p[%dx%d]: Failed!\n", 4 << i, 4 << i);206
+ return false;207
+ }208
+ }209
+ }210
for (int i = 0; i < NUM_TR_SIZE; i++)211
{212
if (opt.cu[i].count_nonzero)213
214
printf("nquant\t\t");215
REPORT_SPEEDUP(opt.nquant, ref.nquant, short_test_buff[0], int_test_buff[1], mshortbuf2, 23, 23785, 32 * 32);216
}217
+218
+ for (int value = 0; value < NUM_TR_SIZE; value++)219
+ {220
+ if (opt.cu[value].nonPsyRdoQuant)221
+ {222
+ ALIGN_VAR_32(int64_t, opt_dest[4 * MAX_TU_SIZE]);223
+ int64_t totalRdCost = 0;224
+ int64_t totalUncodedCost = 0;225
+ printf("nonPsyRdoQuant[%dx%d]", 4 << value, 4 << value);226
+ REPORT_SPEEDUP(opt.cu[value].nonPsyRdoQuant, ref.cu[value].nonPsyRdoQuant, short_test_buff[0], opt_dest, &totalUncodedCost, &totalRdCost, 0);227
+ }228
+ }229
+ for (int value = 0; value < NUM_TR_SIZE; value++)230
+ {231
+ if (opt.cu[value].psyRdoQuant)232
+ {233
+ ALIGN_VAR_32(int64_t, opt_dest[4 * MAX_TU_SIZE]);234
+ int64_t totalRdCost = 0;235
+ int64_t totalUncodedCost = 0;236
+ int64_t *psyScale = X265_MALLOC(int64_t, 1);237
+ *psyScale = 0;238
+ printf("psyRdoQuant[%dx%d]", 4 << value, 4 << value);239
+ REPORT_SPEEDUP(opt.cu[value].psyRdoQuant, ref.cu[value].psyRdoQuant, short_test_buff[0], short_test_buff1[0], opt_dest, &totalUncodedCost, &totalRdCost, psyScale, 0);240
+ }241
+ }242
+ for (int value = 0; value < NUM_TR_SIZE; value++)243
+ {244
+ if (opt.cu[value].psyRdoQuant_1p)245
+ {246
+ ALIGN_VAR_32(int64_t, opt_dest[4 * MAX_TU_SIZE]);247
+ int64_t totalRdCost = 0;248
+ int64_t totalUncodedCost = 0;249
+ printf("psyRdoQuant_1p[%dx%d]", 4 << value, 4 << value);250
+ REPORT_SPEEDUP(opt.cu[value].psyRdoQuant_1p, ref.cu[value].psyRdoQuant_1p, short_test_buff[0], opt_dest, &totalUncodedCost, &totalRdCost, 0);251
+ }252
+ }253
for (int value = 0; value < NUM_TR_SIZE; value++)254
{255
if (opt.cu[value].count_nonzero)256
x265_2.7.tar.gz/source/test/mbdstharness.h -> x265_2.9.tar.gz/source/test/mbdstharness.h
Changed
32
1
2
int mintbuf2[MAX_TU_SIZE];3
int mintbuf3[MAX_TU_SIZE];4
int mintbuf4[MAX_TU_SIZE];5
-6
int16_t short_test_buff[TEST_CASES][TEST_BUF_SIZE];7
+ int16_t short_test_buff1[TEST_CASES][TEST_BUF_SIZE];8
int int_test_buff[TEST_CASES][TEST_BUF_SIZE];9
int int_idct_test_buff[TEST_CASES][TEST_BUF_SIZE];10
-11
uint32_t mubuf1[MAX_TU_SIZE];12
uint32_t mubuf2[MAX_TU_SIZE];13
uint16_t mushortbuf1[MAX_TU_SIZE];14
15
int16_t short_denoise_test_buff1[TEST_CASES][TEST_BUF_SIZE];16
int16_t short_denoise_test_buff2[TEST_CASES][TEST_BUF_SIZE];17
-18
bool check_dequant_primitive(dequant_scaling_t ref, dequant_scaling_t opt);19
bool check_dequant_primitive(dequant_normal_t ref, dequant_normal_t opt);20
+ bool check_nonPsyRdoQuant_primitive(nonPsyRdoQuant_t ref, nonPsyRdoQuant_t opt);21
+ bool check_psyRdoQuant_primitive(psyRdoQuant_t ref, psyRdoQuant_t opt);22
bool check_quant_primitive(quant_t ref, quant_t opt);23
bool check_nquant_primitive(nquant_t ref, nquant_t opt);24
bool check_dct_primitive(dct_t ref, dct_t opt, intptr_t width);25
bool check_idct_primitive(idct_t ref, idct_t opt, intptr_t width);26
bool check_count_nonzero_primitive(count_nonzero_t ref, count_nonzero_t opt);27
bool check_denoise_dct_primitive(denoiseDct_t ref, denoiseDct_t opt);28
+ bool check_psyRdoQuant_primitive_avx2(psyRdoQuant_t1 ref, psyRdoQuant_t1 opt);29
30
public:31
32
x265_2.7.tar.gz/source/test/pixelharness.cpp -> x265_2.9.tar.gz/source/test/pixelharness.cpp
Changed
775
1
2
return true;3
}4
5
+bool PixelHarness::check_calresidual_aligned(calcresidual_t ref, calcresidual_t opt)6
+{7
+ ALIGN_VAR_16(int16_t, ref_dest[64 * 64]);8
+ ALIGN_VAR_16(int16_t, opt_dest[64 * 64]);9
+ memset(ref_dest, 0, 64 * 64 * sizeof(int16_t));10
+ memset(opt_dest, 0, 64 * 64 * sizeof(int16_t));11
+12
+ int j = 0;13
+ intptr_t stride = STRIDE;14
+ for (int i = 0; i < ITERS; i++)15
+ {16
+ int index = i % TEST_CASES;17
+ checked(opt, pbuf1 + j, pixel_test_buff[index] + j, opt_dest, stride);18
+ ref(pbuf1 + j, pixel_test_buff[index] + j, ref_dest, stride);19
+20
+ if (memcmp(ref_dest, opt_dest, 64 * 64 * sizeof(int16_t)))21
+ return false;22
+23
+ reportfail();24
+ j += INCR;25
+ }26
+27
+ return true;28
+}29
+30
bool PixelHarness::check_ssd_s(pixel_ssd_s_t ref, pixel_ssd_s_t opt)31
{32
int j = 0;33
34
reportfail();35
j += INCR;36
}37
-38
return true;39
}40
+bool PixelHarness::check_ssd_s_aligned(pixel_ssd_s_t ref, pixel_ssd_s_t opt)41
+{42
+ int j = 0;43
+ for (int i = 0; i < ITERS; i++)44
+ {45
+ // NOTE: stride must be multiple of 16, because minimum block is 4x446
+ int stride = STRIDE;47
+ sse_t cres = ref(sbuf1 + j, stride);48
+ sse_t vres = (sse_t)checked(opt, sbuf1 + j, (intptr_t)stride);49
+50
+ if (cres != vres)51
+ return false;52
+53
+ reportfail();54
+ j += INCR+32;55
+ }56
57
+ return true;58
+}59
bool PixelHarness::check_weightp(weightp_sp_t ref, weightp_sp_t opt)60
{61
ALIGN_VAR_16(pixel, ref_dest[64 * (64 + 1)]);62
63
memset(ref_dest, 0, 64 * 64 * sizeof(pixel));64
memset(opt_dest, 0, 64 * 64 * sizeof(pixel));65
int j = 0;66
+ bool enableavx512 = true;67
int width = 16 * (rand() % 4 + 1);68
+ int cpuid = X265_NS::cpu_detect(enableavx512);69
+ if (cpuid & X265_CPU_AVX512)70
+ width = 32 * (rand() % 2 + 1);71
int height = 8;72
int w0 = rand() % 128;73
int shift = rand() % 8; // maximum is 7, see setFromWeightAndOffset()74
75
76
return true;77
}78
-79
bool PixelHarness::check_cpy1Dto2D_shl_t(cpy1Dto2D_shl_t ref, cpy1Dto2D_shl_t opt)80
{81
- ALIGN_VAR_16(int16_t, ref_dest[64 * 64]);82
- ALIGN_VAR_16(int16_t, opt_dest[64 * 64]);83
-84
+ ALIGN_VAR_64(int16_t, ref_dest[64 * 64]);85
+ ALIGN_VAR_64(int16_t, opt_dest[64 * 64]);86
memset(ref_dest, 0xCD, sizeof(ref_dest));87
memset(opt_dest, 0xCD, sizeof(opt_dest));88
89
90
91
return true;92
}93
+bool PixelHarness::check_cpy1Dto2D_shl_aligned_t(cpy1Dto2D_shl_t ref, cpy1Dto2D_shl_t opt)94
+{95
+ ALIGN_VAR_64(int16_t, ref_dest[64 * 64]);96
+ ALIGN_VAR_64(int16_t, opt_dest[64 * 64]);97
+98
+ memset(ref_dest, 0xCD, sizeof(ref_dest));99
+ memset(opt_dest, 0xCD, sizeof(opt_dest));100
+101
+ int j = 0;102
+ intptr_t stride = STRIDE;103
+ for (int i = 0; i < ITERS; i++)104
+ {105
+ int shift = (rand() % 7 + 1);106
+107
+ int index = i % TEST_CASES;108
+ checked(opt, opt_dest, short_test_buff[index] + j, stride, shift);109
+ ref(ref_dest, short_test_buff[index] + j, stride, shift);110
+111
+ if (memcmp(ref_dest, opt_dest, 64 * 64 * sizeof(int16_t)))112
+ return false;113
+114
+ reportfail();115
+ j += INCR + 32;116
+ }117
+118
+ return true;119
+}120
121
bool PixelHarness::check_cpy1Dto2D_shr_t(cpy1Dto2D_shr_t ref, cpy1Dto2D_shr_t opt)122
{123
124
125
return true;126
}127
-128
bool PixelHarness::check_pixelavg_pp(pixelavg_pp_t ref, pixelavg_pp_t opt)129
{130
- ALIGN_VAR_16(pixel, ref_dest[64 * 64]);131
- ALIGN_VAR_16(pixel, opt_dest[64 * 64]);132
+ ALIGN_VAR_64(pixel, ref_dest[64 * 64]);133
+ ALIGN_VAR_64(pixel, opt_dest[64 * 64]);134
+ int j = 0;135
+ memset(ref_dest, 0xCD, sizeof(ref_dest));136
+ memset(opt_dest, 0xCD, sizeof(opt_dest));137
+138
+ intptr_t stride = STRIDE;139
+ for (int i = 0; i < ITERS; i++)140
+ {141
+ int index1 = rand() % TEST_CASES;142
+ int index2 = rand() % TEST_CASES;143
+ checked(ref, ref_dest, stride, pixel_test_buff[index1] + j,144
+ stride, pixel_test_buff[index2] + j, stride, 32);145
+ opt(opt_dest, stride, pixel_test_buff[index1] + j,146
+ stride, pixel_test_buff[index2] + j, stride, 32);147
+148
+ if (memcmp(ref_dest, opt_dest, 64 * 64 * sizeof(pixel)))149
+ return false;150
+151
+ reportfail();152
+ j += INCR;153
+ }154
+155
+ return true;156
+}157
+bool PixelHarness::check_pixelavg_pp_aligned(pixelavg_pp_t ref, pixelavg_pp_t opt)158
+{159
+ ALIGN_VAR_64(pixel, ref_dest[64 * 64]);160
+ ALIGN_VAR_64(pixel, opt_dest[64 * 64]);161
162
int j = 0;163
164
165
return false;166
167
reportfail();168
- j += INCR;169
+ j += INCR + 32;170
}171
172
return true;173
174
175
bool PixelHarness::check_blockfill_s(blockfill_s_t ref, blockfill_s_t opt)176
{177
- ALIGN_VAR_16(int16_t, ref_dest[64 * 64]);178
- ALIGN_VAR_16(int16_t, opt_dest[64 * 64]);179
+ ALIGN_VAR_64(int16_t, ref_dest[64 * 64]);180
+ ALIGN_VAR_64(int16_t, opt_dest[64 * 64]);181
+182
+ memset(ref_dest, 0xCD, sizeof(ref_dest));183
+ memset(opt_dest, 0xCD, sizeof(opt_dest));184
+185
+ intptr_t stride = 64;186
+ for (int i = 0; i < ITERS; i++)187
+ {188
+ int16_t value = (rand() % SHORT_MAX) + 1;189
+190
+ checked(opt, opt_dest, stride, value);191
+ ref(ref_dest, stride, value);192
+193
+ if (memcmp(ref_dest, opt_dest, 64 * 64 * sizeof(int16_t)))194
+ return false;195
+196
+ reportfail();197
+ }198
+199
+ return true;200
+}201
+202
+bool PixelHarness::check_blockfill_s_aligned(blockfill_s_t ref, blockfill_s_t opt)203
+{204
+ ALIGN_VAR_64(int16_t, ref_dest[64 * 64]);205
+ ALIGN_VAR_64(int16_t, opt_dest[64 * 64]);206
207
memset(ref_dest, 0xCD, sizeof(ref_dest));208
memset(opt_dest, 0xCD, sizeof(opt_dest));209
210
211
bool PixelHarness::check_scale1D_pp(scale1D_t ref, scale1D_t opt)212
{213
- ALIGN_VAR_16(pixel, ref_dest[64 * 64]);214
- ALIGN_VAR_16(pixel, opt_dest[64 * 64]);215
+ ALIGN_VAR_64(pixel, ref_dest[64 * 64]);216
+ ALIGN_VAR_64(pixel, opt_dest[64 * 64]);217
218
memset(ref_dest, 0, sizeof(ref_dest));219
memset(opt_dest, 0, sizeof(opt_dest));220
221
return true;222
}223
224
+bool PixelHarness::check_scale1D_pp_aligned(scale1D_t ref, scale1D_t opt)225
+{226
+ ALIGN_VAR_64(pixel, ref_dest[64 * 64]);227
+ ALIGN_VAR_64(pixel, opt_dest[64 * 64]);228
+229
+ memset(ref_dest, 0, sizeof(ref_dest));230
+ memset(opt_dest, 0, sizeof(opt_dest));231
+232
+ int j = 0;233
+ for (int i = 0; i < ITERS; i++)234
+ {235
+ int index = i % TEST_CASES;236
+ checked(opt, opt_dest, pixel_test_buff[index] + j);237
+ ref(ref_dest, pixel_test_buff[index] + j);238
+239
+ if (memcmp(ref_dest, opt_dest, 64 * 64 * sizeof(pixel)))240
+ return false;241
+242
+ reportfail();243
+ j += INCR * 2;244
+ }245
+246
+ return true;247
+}248
+249
bool PixelHarness::check_scale2D_pp(scale2D_t ref, scale2D_t opt)250
{251
ALIGN_VAR_16(pixel, ref_dest[64 * 64]);252
253
return true;254
}255
256
+bool PixelHarness::check_pixel_add_ps_aligned(pixel_add_ps_t ref, pixel_add_ps_t opt)257
+{258
+ ALIGN_VAR_64(pixel, ref_dest[64 * 64]);259
+ ALIGN_VAR_64(pixel, opt_dest[64 * 64]);260
+261
+ memset(ref_dest, 0xCD, sizeof(ref_dest));262
+ memset(opt_dest, 0xCD, sizeof(opt_dest));263
+264
+ int j = 0;265
+ intptr_t stride2 = 64, stride = STRIDE;266
+ for (int i = 0; i < ITERS; i++)267
+ {268
+ int index1 = rand() % TEST_CASES;269
+ int index2 = rand() % TEST_CASES;270
+ checked(opt, opt_dest, stride2, pixel_test_buff[index1] + j, short_test_buff[index2] + j, stride, stride);271
+ ref(ref_dest, stride2, pixel_test_buff[index1] + j, short_test_buff[index2] + j, stride, stride);272
+ if (memcmp(ref_dest, opt_dest, 64 * 64 * sizeof(pixel)))273
+ return false;274
+275
+ reportfail();276
+ j += 2 * INCR;277
+ }278
+ return true;279
+}280
+281
bool PixelHarness::check_pixel_var(var_t ref, var_t opt)282
{283
int j = 0;284
285
286
bool PixelHarness::check_addAvg(addAvg_t ref, addAvg_t opt)287
{288
- ALIGN_VAR_16(pixel, ref_dest[64 * 64]);289
- ALIGN_VAR_16(pixel, opt_dest[64 * 64]);290
+ ALIGN_VAR_64(pixel, ref_dest[64 * 64]);291
+ ALIGN_VAR_64(pixel, opt_dest[64 * 64]);292
293
int j = 0;294
295
296
return true;297
}298
299
+bool PixelHarness::check_addAvg_aligned(addAvg_t ref, addAvg_t opt)300
+{301
+ ALIGN_VAR_64(pixel, ref_dest[64 * 64]);302
+ ALIGN_VAR_64(pixel, opt_dest[64 * 64]);303
+304
+ int j = 0;305
+306
+ memset(ref_dest, 0xCD, sizeof(ref_dest));307
+ memset(opt_dest, 0xCD, sizeof(opt_dest));308
+ intptr_t stride = STRIDE;309
+310
+ for (int i = 0; i < ITERS; i++)311
+ {312
+ int index1 = rand() % TEST_CASES;313
+ int index2 = rand() % TEST_CASES;314
+ ref(short_test_buff2[index1] + j, short_test_buff2[index2] + j, ref_dest, stride, stride, stride);315
+ checked(opt, short_test_buff2[index1] + j, short_test_buff2[index2] + j, opt_dest, stride, stride, stride);316
+ if (memcmp(ref_dest, opt_dest, 64 * 64 * sizeof(pixel)))317
+ return false;318
+319
+ reportfail();320
+ j += INCR * 2;321
+ }322
+323
+ return true;324
+}325
bool PixelHarness::check_calSign(sign_t ref, sign_t opt)326
{327
ALIGN_VAR_16(int8_t, ref_dest[64 * 2]);328
329
return false;330
}331
}332
-333
- if (opt.pu[part].pixelavg_pp)334
+ if (opt.pu[part].pixelavg_pp[NONALIGNED])335
{336
- if (!check_pixelavg_pp(ref.pu[part].pixelavg_pp, opt.pu[part].pixelavg_pp))337
+ if (!check_pixelavg_pp(ref.pu[part].pixelavg_pp[NONALIGNED], opt.pu[part].pixelavg_pp[NONALIGNED]))338
{339
printf("pixelavg_pp[%s]: failed!\n", lumaPartStr[part]);340
return false;341
}342
}343
+ if (opt.pu[part].pixelavg_pp[ALIGNED])344
+ {345
+ if (!check_pixelavg_pp_aligned(ref.pu[part].pixelavg_pp[ALIGNED], opt.pu[part].pixelavg_pp[ALIGNED]))346
+ {347
+ printf("pixelavg_pp_aligned[%s]: failed!\n", lumaPartStr[part]);348
+ return false;349
+ }350
+ }351
352
if (opt.pu[part].copy_pp)353
{354
355
}356
}357
358
- if (opt.pu[part].addAvg)359
+ if (opt.pu[part].addAvg[NONALIGNED])360
{361
- if (!check_addAvg(ref.pu[part].addAvg, opt.pu[part].addAvg))362
+ if (!check_addAvg(ref.pu[part].addAvg[NONALIGNED], opt.pu[part].addAvg[NONALIGNED]))363
{364
printf("addAvg[%s] failed\n", lumaPartStr[part]);365
return false;366
}367
}368
369
+ if (opt.pu[part].addAvg[ALIGNED])370
+ {371
+ if (!check_addAvg_aligned(ref.pu[part].addAvg[ALIGNED], opt.pu[part].addAvg[ALIGNED]))372
+ {373
+ printf("addAvg_aligned[%s] failed\n", lumaPartStr[part]);374
+ return false;375
+ }376
+ }377
+378
if (part < NUM_CU_SIZES)379
{380
if (opt.cu[part].sse_pp)381
382
}383
}384
385
- if (opt.cu[part].add_ps)386
+ if (opt.cu[part].add_ps[NONALIGNED])387
{388
- if (!check_pixel_add_ps(ref.cu[part].add_ps, opt.cu[part].add_ps))389
+ if (!check_pixel_add_ps(ref.cu[part].add_ps[NONALIGNED], opt.cu[part].add_ps[NONALIGNED]))390
{391
printf("add_ps[%s] failed\n", lumaPartStr[part]);392
return false;393
}394
}395
396
+ if (opt.cu[part].add_ps[ALIGNED])397
+ {398
+ if (!check_pixel_add_ps_aligned(ref.cu[part].add_ps[ALIGNED], opt.cu[part].add_ps[ALIGNED]))399
+ {400
+ printf("add_ps_aligned[%s] failed\n", lumaPartStr[part]);401
+ return false;402
+ }403
+ }404
+405
if (opt.cu[part].copy_ss)406
{407
if (!check_copy_ss(ref.cu[part].copy_ss, opt.cu[part].copy_ss))408
409
return false;410
}411
}412
- if (opt.chroma[i].pu[part].addAvg)413
+ if (opt.chroma[i].pu[part].addAvg[NONALIGNED])414
{415
- if (!check_addAvg(ref.chroma[i].pu[part].addAvg, opt.chroma[i].pu[part].addAvg))416
+ if (!check_addAvg(ref.chroma[i].pu[part].addAvg[NONALIGNED], opt.chroma[i].pu[part].addAvg[NONALIGNED]))417
{418
printf("chroma_addAvg[%s][%s] failed\n", x265_source_csp_names[i], chromaPartStr[i][part]);419
return false;420
}421
}422
+ if (opt.chroma[i].pu[part].addAvg[ALIGNED])423
+ {424
+ if (!check_addAvg_aligned(ref.chroma[i].pu[part].addAvg[ALIGNED], opt.chroma[i].pu[part].addAvg[ALIGNED]))425
+ {426
+ printf("chroma_addAvg_aligned[%s][%s] failed\n", x265_source_csp_names[i], chromaPartStr[i][part]);427
+ return false;428
+ }429
+ }430
if (opt.chroma[i].pu[part].satd)431
{432
if (!check_pixelcmp(ref.chroma[i].pu[part].satd, opt.chroma[i].pu[part].satd))433
434
return false;435
}436
}437
- if (opt.chroma[i].cu[part].add_ps)438
+ if (opt.chroma[i].cu[part].add_ps[NONALIGNED])439
{440
- if (!check_pixel_add_ps(ref.chroma[i].cu[part].add_ps, opt.chroma[i].cu[part].add_ps))441
+ if (!check_pixel_add_ps(ref.chroma[i].cu[part].add_ps[NONALIGNED], opt.chroma[i].cu[part].add_ps[NONALIGNED]))442
{443
printf("chroma_add_ps[%s][%s] failed\n", x265_source_csp_names[i], chromaPartStr[i][part]);444
return false;445
}446
}447
+ if (opt.chroma[i].cu[part].add_ps[ALIGNED])448
+ {449
+ if (!check_pixel_add_ps_aligned(ref.chroma[i].cu[part].add_ps[ALIGNED], opt.chroma[i].cu[part].add_ps[ALIGNED]))450
+ {451
+ printf("chroma_add_ps_aligned[%s][%s] failed\n", x265_source_csp_names[i], chromaPartStr[i][part]);452
+ return false;453
+ }454
+ }455
if (opt.chroma[i].cu[part].copy_sp)456
{457
if (!check_copy_sp(ref.chroma[i].cu[part].copy_sp, opt.chroma[i].cu[part].copy_sp))458
459
}460
}461
462
- if (opt.cu[i].blockfill_s)463
+ if (opt.cu[i].blockfill_s[NONALIGNED])464
{465
- if (!check_blockfill_s(ref.cu[i].blockfill_s, opt.cu[i].blockfill_s))466
+ if (!check_blockfill_s(ref.cu[i].blockfill_s[NONALIGNED], opt.cu[i].blockfill_s[NONALIGNED]))467
{468
printf("blockfill_s[%dx%d]: failed!\n", 4 << i, 4 << i);469
return false;470
}471
}472
473
+ if (opt.cu[i].blockfill_s[ALIGNED])474
+ {475
+ if (!check_blockfill_s_aligned(ref.cu[i].blockfill_s[ALIGNED], opt.cu[i].blockfill_s[ALIGNED]))476
+ {477
+ printf("blockfill_s_aligned[%dx%d]: failed!\n", 4 << i, 4 << i);478
+ return false;479
+ }480
+ }481
if (opt.cu[i].var)482
{483
if (!check_pixel_var(ref.cu[i].var, opt.cu[i].var))484
485
{486
/* TU only primitives */487
488
- if (opt.cu[i].calcresidual)489
+ if (opt.cu[i].calcresidual[NONALIGNED])490
{491
- if (!check_calresidual(ref.cu[i].calcresidual, opt.cu[i].calcresidual))492
+ if (!check_calresidual(ref.cu[i].calcresidual[NONALIGNED], opt.cu[i].calcresidual[NONALIGNED]))493
{494
printf("calcresidual width: %d failed!\n", 4 << i);495
return false;496
}497
}498
499
+ if (opt.cu[i].calcresidual[ALIGNED])500
+ {501
+ if (!check_calresidual_aligned(ref.cu[i].calcresidual[ALIGNED], opt.cu[i].calcresidual[ALIGNED]))502
+ {503
+ printf("calcresidual_aligned width: %d failed!\n", 4 << i);504
+ return false;505
+ }506
+ }507
+508
if (opt.cu[i].transpose)509
{510
if (!check_transpose(ref.cu[i].transpose, opt.cu[i].transpose))511
512
return false;513
}514
}515
-516
- if (opt.cu[i].ssd_s)517
+ if (opt.cu[i].ssd_s[NONALIGNED])518
{519
- if (!check_ssd_s(ref.cu[i].ssd_s, opt.cu[i].ssd_s))520
+ if (!check_ssd_s(ref.cu[i].ssd_s[NONALIGNED], opt.cu[i].ssd_s[NONALIGNED]))521
{522
printf("ssd_s[%dx%d]: failed!\n", 4 << i, 4 << i);523
return false;524
}525
}526
-527
+ if (opt.cu[i].ssd_s[ALIGNED])528
+ {529
+ if (!check_ssd_s_aligned(ref.cu[i].ssd_s[ALIGNED], opt.cu[i].ssd_s[ALIGNED]))530
+ {531
+ printf("ssd_s_aligned[%dx%d]: failed!\n", 4 << i, 4 << i);532
+ return false;533
+ }534
+ }535
if (opt.cu[i].copy_cnt)536
{537
if (!check_copy_cnt_t(ref.cu[i].copy_cnt, opt.cu[i].copy_cnt))538
539
return false;540
}541
}542
-543
- if (opt.cu[i].cpy1Dto2D_shl)544
+ if (opt.cu[i].cpy1Dto2D_shl[NONALIGNED])545
{546
- if (!check_cpy1Dto2D_shl_t(ref.cu[i].cpy1Dto2D_shl, opt.cu[i].cpy1Dto2D_shl))547
+ if (!check_cpy1Dto2D_shl_t(ref.cu[i].cpy1Dto2D_shl[NONALIGNED], opt.cu[i].cpy1Dto2D_shl[NONALIGNED]))548
{549
printf("cpy1Dto2D_shl[%dx%d] failed!\n", 4 << i, 4 << i);550
return false;551
}552
}553
+ if (opt.cu[i].cpy1Dto2D_shl[ALIGNED])554
+ {555
+ if (!check_cpy1Dto2D_shl_aligned_t(ref.cu[i].cpy1Dto2D_shl[ALIGNED], opt.cu[i].cpy1Dto2D_shl[ALIGNED]))556
+ {557
+ printf("cpy1Dto2D_shl_aligned[%dx%d] failed!\n", 4 << i, 4 << i);558
+ return false;559
+ }560
+ }561
562
if (opt.cu[i].cpy1Dto2D_shr)563
{564
565
}566
}567
568
- if (opt.scale1D_128to64)569
+ if (opt.scale1D_128to64[NONALIGNED])570
{571
- if (!check_scale1D_pp(ref.scale1D_128to64, opt.scale1D_128to64))572
+ if (!check_scale1D_pp(ref.scale1D_128to64[NONALIGNED], opt.scale1D_128to64[NONALIGNED]))573
{574
printf("scale1D_128to64 failed!\n");575
return false;576
}577
}578
579
+ if (opt.scale1D_128to64[ALIGNED])580
+ {581
+ if (!check_scale1D_pp_aligned(ref.scale1D_128to64[ALIGNED], opt.scale1D_128to64[ALIGNED]))582
+ {583
+ printf("scale1D_128to64_aligned failed!\n");584
+ return false;585
+ }586
+ }587
+588
if (opt.scale2D_64to32)589
{590
if (!check_scale2D_pp(ref.scale2D_64to32, opt.scale2D_64to32))591
592
HEADER("satd[%s]", lumaPartStr[part]);593
REPORT_SPEEDUP(opt.pu[part].satd, ref.pu[part].satd, pbuf1, STRIDE, fref, STRIDE);594
}595
-596
- if (opt.pu[part].pixelavg_pp)597
+ if (opt.pu[part].pixelavg_pp[NONALIGNED])598
{599
HEADER("avg_pp[%s]", lumaPartStr[part]);600
- REPORT_SPEEDUP(opt.pu[part].pixelavg_pp, ref.pu[part].pixelavg_pp, pbuf1, STRIDE, pbuf2, STRIDE, pbuf3, STRIDE, 32);601
+ REPORT_SPEEDUP(opt.pu[part].pixelavg_pp[NONALIGNED], ref.pu[part].pixelavg_pp[NONALIGNED], pbuf1, STRIDE, pbuf2, STRIDE, pbuf3, STRIDE, 32);602
}603
604
+ if (opt.pu[part].pixelavg_pp[ALIGNED])605
+ {606
+ HEADER("avg_pp_aligned[%s]", lumaPartStr[part]);607
+ REPORT_SPEEDUP(opt.pu[part].pixelavg_pp[ALIGNED], ref.pu[part].pixelavg_pp[ALIGNED], pbuf1, STRIDE, pbuf2, STRIDE, pbuf3, STRIDE, 32);608
+ }609
if (opt.pu[part].sad)610
{611
HEADER("sad[%s]", lumaPartStr[part]);612
613
REPORT_SPEEDUP(opt.pu[part].copy_pp, ref.pu[part].copy_pp, pbuf1, 64, pbuf2, 64);614
}615
616
- if (opt.pu[part].addAvg)617
+ if (opt.pu[part].addAvg[NONALIGNED])618
{619
HEADER("addAvg[%s]", lumaPartStr[part]);620
- REPORT_SPEEDUP(opt.pu[part].addAvg, ref.pu[part].addAvg, sbuf1, sbuf2, pbuf1, STRIDE, STRIDE, STRIDE);621
+ REPORT_SPEEDUP(opt.pu[part].addAvg[NONALIGNED], ref.pu[part].addAvg[NONALIGNED], sbuf1, sbuf2, pbuf1, STRIDE, STRIDE, STRIDE);622
+ }623
+ if (opt.pu[part].addAvg[ALIGNED])624
+ {625
+ HEADER("addAvg_aligned[%s]", lumaPartStr[part]);626
+ REPORT_SPEEDUP(opt.pu[part].addAvg[ALIGNED], ref.pu[part].addAvg[ALIGNED], sbuf1, sbuf2, pbuf1, STRIDE, STRIDE, STRIDE);627
}628
629
if (part < NUM_CU_SIZES)630
631
HEADER("sub_ps[%s]", lumaPartStr[part]);632
REPORT_SPEEDUP(opt.cu[part].sub_ps, ref.cu[part].sub_ps, (int16_t*)pbuf1, FENC_STRIDE, pbuf2, pbuf1, STRIDE, STRIDE);633
}634
- if (opt.cu[part].add_ps)635
+ if (opt.cu[part].add_ps[NONALIGNED])636
{637
HEADER("add_ps[%s]", lumaPartStr[part]);638
- REPORT_SPEEDUP(opt.cu[part].add_ps, ref.cu[part].add_ps, pbuf1, FENC_STRIDE, pbuf2, sbuf1, STRIDE, STRIDE);639
+ REPORT_SPEEDUP(opt.cu[part].add_ps[NONALIGNED], ref.cu[part].add_ps[NONALIGNED], pbuf1, FENC_STRIDE, pbuf2, sbuf1, STRIDE, STRIDE);640
+ }641
+ if (opt.cu[part].add_ps[ALIGNED])642
+ {643
+ HEADER("add_ps_aligned[%s]", lumaPartStr[part]);644
+ REPORT_SPEEDUP(opt.cu[part].add_ps[ALIGNED], ref.cu[part].add_ps[ALIGNED], pbuf1, FENC_STRIDE, pbuf2, sbuf1, STRIDE, STRIDE);645
}646
if (opt.cu[part].copy_ss)647
{648
649
HEADER("[%s] copy_pp[%s]", x265_source_csp_names[i], chromaPartStr[i][part]);650
REPORT_SPEEDUP(opt.chroma[i].pu[part].copy_pp, ref.chroma[i].pu[part].copy_pp, pbuf1, 64, pbuf2, 128);651
}652
- if (opt.chroma[i].pu[part].addAvg)653
+ if (opt.chroma[i].pu[part].addAvg[NONALIGNED])654
{655
HEADER("[%s] addAvg[%s]", x265_source_csp_names[i], chromaPartStr[i][part]);656
- REPORT_SPEEDUP(opt.chroma[i].pu[part].addAvg, ref.chroma[i].pu[part].addAvg, sbuf1, sbuf2, pbuf1, STRIDE, STRIDE, STRIDE);657
+ REPORT_SPEEDUP(opt.chroma[i].pu[part].addAvg[NONALIGNED], ref.chroma[i].pu[part].addAvg[NONALIGNED], sbuf1, sbuf2, pbuf1, STRIDE, STRIDE, STRIDE);658
+ }659
+ if (opt.chroma[i].pu[part].addAvg[ALIGNED])660
+ {661
+ HEADER("[%s] addAvg_aligned[%s]", x265_source_csp_names[i], chromaPartStr[i][part]);662
+ REPORT_SPEEDUP(opt.chroma[i].pu[part].addAvg[ALIGNED], ref.chroma[i].pu[part].addAvg[ALIGNED], sbuf1, sbuf2, pbuf1, STRIDE, STRIDE, STRIDE);663
}664
if (opt.chroma[i].pu[part].satd)665
{666
667
HEADER("[%s] sub_ps[%s]", x265_source_csp_names[i], chromaPartStr[i][part]);668
REPORT_SPEEDUP(opt.chroma[i].cu[part].sub_ps, ref.chroma[i].cu[part].sub_ps, (int16_t*)pbuf1, FENC_STRIDE, pbuf2, pbuf1, STRIDE, STRIDE);669
}670
- if (opt.chroma[i].cu[part].add_ps)671
+ if (opt.chroma[i].cu[part].add_ps[NONALIGNED])672
{673
HEADER("[%s] add_ps[%s]", x265_source_csp_names[i], chromaPartStr[i][part]);674
- REPORT_SPEEDUP(opt.chroma[i].cu[part].add_ps, ref.chroma[i].cu[part].add_ps, pbuf1, FENC_STRIDE, pbuf2, sbuf1, STRIDE, STRIDE);675
+ REPORT_SPEEDUP(opt.chroma[i].cu[part].add_ps[NONALIGNED], ref.chroma[i].cu[part].add_ps[NONALIGNED], pbuf1, FENC_STRIDE, pbuf2, sbuf1, STRIDE, STRIDE);676
+ }677
+ if (opt.chroma[i].cu[part].add_ps[ALIGNED])678
+ {679
+ HEADER("[%s] add_ps_aligned[%s]", x265_source_csp_names[i], chromaPartStr[i][part]);680
+ REPORT_SPEEDUP(opt.chroma[i].cu[part].add_ps[ALIGNED], ref.chroma[i].cu[part].add_ps[ALIGNED], pbuf1, FENC_STRIDE, pbuf2, sbuf1, STRIDE, STRIDE);681
}682
if (opt.chroma[i].cu[part].sa8d)683
{684
685
measurePartition(part, ref, opt);686
}687
}688
-689
for (int i = 0; i < NUM_CU_SIZES; i++)690
{691
- if ((i <= BLOCK_32x32) && opt.cu[i].ssd_s)692
+ if ((i <= BLOCK_32x32) && opt.cu[i].ssd_s[NONALIGNED])693
{694
HEADER("ssd_s[%dx%d]", 4 << i, 4 << i);695
- REPORT_SPEEDUP(opt.cu[i].ssd_s, ref.cu[i].ssd_s, sbuf1, STRIDE);696
+ REPORT_SPEEDUP(opt.cu[i].ssd_s[NONALIGNED], ref.cu[i].ssd_s[NONALIGNED], sbuf1, STRIDE);697
+ }698
+ if ((i <= BLOCK_32x32) && opt.cu[i].ssd_s[ALIGNED])699
+ {700
+ HEADER("ssd_s_aligned[%dx%d]", 4 << i, 4 << i);701
+ REPORT_SPEEDUP(opt.cu[i].ssd_s[ALIGNED], ref.cu[i].ssd_s[ALIGNED], sbuf1, STRIDE);702
}703
if (opt.cu[i].sa8d)704
{705
HEADER("sa8d[%dx%d]", 4 << i, 4 << i);706
REPORT_SPEEDUP(opt.cu[i].sa8d, ref.cu[i].sa8d, pbuf1, STRIDE, pbuf2, STRIDE);707
}708
- if (opt.cu[i].calcresidual)709
+ if (opt.cu[i].calcresidual[NONALIGNED])710
{711
HEADER("residual[%dx%d]", 4 << i, 4 << i);712
- REPORT_SPEEDUP(opt.cu[i].calcresidual, ref.cu[i].calcresidual, pbuf1, pbuf2, sbuf1, 64);713
+ REPORT_SPEEDUP(opt.cu[i].calcresidual[NONALIGNED], ref.cu[i].calcresidual[NONALIGNED], pbuf1, pbuf2, sbuf1, 64);714
}715
-716
- if (opt.cu[i].blockfill_s)717
+ if (opt.cu[i].calcresidual[ALIGNED])718
+ {719
+ HEADER("residual_aligned[%dx%d]", 4 << i, 4 << i);720
+ REPORT_SPEEDUP(opt.cu[i].calcresidual[ALIGNED], ref.cu[i].calcresidual[ALIGNED], pbuf1, pbuf2, sbuf1, 64);721
+ }722
+ if (opt.cu[i].blockfill_s[NONALIGNED])723
{724
HEADER("blkfill[%dx%d]", 4 << i, 4 << i);725
- REPORT_SPEEDUP(opt.cu[i].blockfill_s, ref.cu[i].blockfill_s, sbuf1, 64, SHORT_MAX);726
+ REPORT_SPEEDUP(opt.cu[i].blockfill_s[NONALIGNED], ref.cu[i].blockfill_s[NONALIGNED], sbuf1, 64, SHORT_MAX);727
+ }728
+ if (opt.cu[i].blockfill_s[ALIGNED])729
+ {730
+ HEADER("blkfill_aligned[%dx%d]", 4 << i, 4 << i);731
+ REPORT_SPEEDUP(opt.cu[i].blockfill_s[ALIGNED], ref.cu[i].blockfill_s[ALIGNED], sbuf1, 64, SHORT_MAX);732
}733
734
if (opt.cu[i].transpose)735
736
HEADER("cpy2Dto1D_shr[%dx%d]", 4 << i, 4 << i);737
REPORT_SPEEDUP(opt.cu[i].cpy2Dto1D_shr, ref.cu[i].cpy2Dto1D_shr, sbuf1, sbuf2, STRIDE, 3);738
}739
-740
- if ((i < BLOCK_64x64) && opt.cu[i].cpy1Dto2D_shl)741
+ if ((i < BLOCK_64x64) && opt.cu[i].cpy1Dto2D_shl[NONALIGNED])742
{743
HEADER("cpy1Dto2D_shl[%dx%d]", 4 << i, 4 << i);744
- REPORT_SPEEDUP(opt.cu[i].cpy1Dto2D_shl, ref.cu[i].cpy1Dto2D_shl, sbuf1, sbuf2, STRIDE, 64);745
+ REPORT_SPEEDUP(opt.cu[i].cpy1Dto2D_shl[NONALIGNED], ref.cu[i].cpy1Dto2D_shl[NONALIGNED], sbuf1, sbuf2, STRIDE, 64);746
}747
748
+ if ((i < BLOCK_64x64) && opt.cu[i].cpy1Dto2D_shl[ALIGNED])749
+ {750
+ HEADER("cpy1Dto2D_shl_aligned[%dx%d]", 4 << i, 4 << i);751
+ REPORT_SPEEDUP(opt.cu[i].cpy1Dto2D_shl[ALIGNED], ref.cu[i].cpy1Dto2D_shl[ALIGNED], sbuf1, sbuf2, STRIDE, 64);752
+ }753
if ((i < BLOCK_64x64) && opt.cu[i].cpy1Dto2D_shr)754
{755
HEADER("cpy1Dto2D_shr[%dx%d]", 4 << i, 4 << i);756
757
REPORT_SPEEDUP(opt.frameInitLowres, ref.frameInitLowres, pbuf2, pbuf1, pbuf2, pbuf3, pbuf4, 64, 64, 64, 64);758
}759
760
- if (opt.scale1D_128to64)761
+ if (opt.scale1D_128to64[NONALIGNED])762
{763
HEADER0("scale1D_128to64");764
- REPORT_SPEEDUP(opt.scale1D_128to64, ref.scale1D_128to64, pbuf2, pbuf1);765
+ REPORT_SPEEDUP(opt.scale1D_128to64[NONALIGNED], ref.scale1D_128to64[NONALIGNED], pbuf2, pbuf1);766
+ }767
+768
+ if (opt.scale1D_128to64[ALIGNED])769
+ {770
+ HEADER0("scale1D_128to64_aligned");771
+ REPORT_SPEEDUP(opt.scale1D_128to64[ALIGNED], ref.scale1D_128to64[ALIGNED], pbuf2, pbuf1);772
}773
774
if (opt.scale2D_64to32)775
x265_2.7.tar.gz/source/test/pixelharness.h -> x265_2.9.tar.gz/source/test/pixelharness.h
Changed
89
1
2
enum { RMAX = PIXEL_MAX - PIXEL_MIN }; //The maximum value obtained by subtracting pixel values (residual max)3
enum { RMIN = PIXEL_MIN - PIXEL_MAX }; //The minimum value obtained by subtracting pixel values (residual min)4
5
- ALIGN_VAR_32(pixel, pbuf1[BUFFSIZE]);6
- pixel pbuf2[BUFFSIZE];7
- pixel pbuf3[BUFFSIZE];8
- pixel pbuf4[BUFFSIZE];9
- int ibuf1[BUFFSIZE];10
- int8_t psbuf1[BUFFSIZE];11
- int8_t psbuf2[BUFFSIZE];12
- int8_t psbuf3[BUFFSIZE];13
- int8_t psbuf4[BUFFSIZE];14
- int8_t psbuf5[BUFFSIZE];15
+ ALIGN_VAR_64(pixel, pbuf1[BUFFSIZE]);16
+ ALIGN_VAR_64(pixel, pbuf2[BUFFSIZE]);17
+ ALIGN_VAR_64(pixel, pbuf3[BUFFSIZE]);18
+ ALIGN_VAR_64(pixel, pbuf4[BUFFSIZE]);19
+ ALIGN_VAR_64(int, ibuf1[BUFFSIZE]);20
+ ALIGN_VAR_64(int8_t, psbuf1[BUFFSIZE]);21
+ ALIGN_VAR_64(int8_t, psbuf2[BUFFSIZE]);22
+ ALIGN_VAR_64(int8_t, psbuf3[BUFFSIZE]);23
+ ALIGN_VAR_64(int8_t, psbuf4[BUFFSIZE]);24
+ ALIGN_VAR_64(int8_t, psbuf5[BUFFSIZE]);25
26
- int16_t sbuf1[BUFFSIZE];27
- int16_t sbuf2[BUFFSIZE];28
- int16_t sbuf3[BUFFSIZE];29
+ ALIGN_VAR_64(int16_t, sbuf1[BUFFSIZE]);30
+ ALIGN_VAR_64(int16_t, sbuf2[BUFFSIZE]);31
+ ALIGN_VAR_64(int16_t, sbuf3[BUFFSIZE]);32
33
- pixel pixel_test_buff[TEST_CASES][BUFFSIZE];34
- int16_t short_test_buff[TEST_CASES][BUFFSIZE];35
- int16_t short_test_buff1[TEST_CASES][BUFFSIZE];36
- int16_t short_test_buff2[TEST_CASES][BUFFSIZE];37
- int int_test_buff[TEST_CASES][BUFFSIZE];38
- uint16_t ushort_test_buff[TEST_CASES][BUFFSIZE];39
- uint8_t uchar_test_buff[TEST_CASES][BUFFSIZE];40
- double double_test_buff[TEST_CASES][BUFFSIZE];41
- int16_t residual_test_buff[TEST_CASES][BUFFSIZE];42
+ ALIGN_VAR_64(pixel, pixel_test_buff[TEST_CASES][BUFFSIZE]);43
+ ALIGN_VAR_64(int16_t, short_test_buff[TEST_CASES][BUFFSIZE]);44
+ ALIGN_VAR_64(int16_t, short_test_buff1[TEST_CASES][BUFFSIZE]);45
+ ALIGN_VAR_64(int16_t, short_test_buff2[TEST_CASES][BUFFSIZE]);46
+ ALIGN_VAR_64(int, int_test_buff[TEST_CASES][BUFFSIZE]);47
+ ALIGN_VAR_64(uint16_t, ushort_test_buff[TEST_CASES][BUFFSIZE]);48
+ ALIGN_VAR_64(uint8_t, uchar_test_buff[TEST_CASES][BUFFSIZE]);49
+ ALIGN_VAR_64(double, double_test_buff[TEST_CASES][BUFFSIZE]);50
+ ALIGN_VAR_64(int16_t, residual_test_buff[TEST_CASES][BUFFSIZE]);51
52
bool check_pixelcmp(pixelcmp_t ref, pixelcmp_t opt);53
bool check_pixel_sse(pixel_sse_t ref, pixel_sse_t opt);54
55
bool check_copy_ps(copy_ps_t ref, copy_ps_t opt);56
bool check_copy_ss(copy_ss_t ref, copy_ss_t opt);57
bool check_pixelavg_pp(pixelavg_pp_t ref, pixelavg_pp_t opt);58
+ bool check_pixelavg_pp_aligned(pixelavg_pp_t ref, pixelavg_pp_t opt);59
bool check_pixel_sub_ps(pixel_sub_ps_t ref, pixel_sub_ps_t opt);60
bool check_pixel_add_ps(pixel_add_ps_t ref, pixel_add_ps_t opt);61
+ bool check_pixel_add_ps_aligned(pixel_add_ps_t ref, pixel_add_ps_t opt);62
bool check_scale1D_pp(scale1D_t ref, scale1D_t opt);63
+ bool check_scale1D_pp_aligned(scale1D_t ref, scale1D_t opt);64
bool check_scale2D_pp(scale2D_t ref, scale2D_t opt);65
bool check_ssd_s(pixel_ssd_s_t ref, pixel_ssd_s_t opt);66
+ bool check_ssd_s_aligned(pixel_ssd_s_t ref, pixel_ssd_s_t opt);67
bool check_blockfill_s(blockfill_s_t ref, blockfill_s_t opt);68
+ bool check_blockfill_s_aligned(blockfill_s_t ref, blockfill_s_t opt);69
bool check_calresidual(calcresidual_t ref, calcresidual_t opt);70
+ bool check_calresidual_aligned(calcresidual_t ref, calcresidual_t opt);71
bool check_transpose(transpose_t ref, transpose_t opt);72
bool check_weightp(weightp_pp_t ref, weightp_pp_t opt);73
bool check_weightp(weightp_sp_t ref, weightp_sp_t opt);74
75
bool check_cpy2Dto1D_shl_t(cpy2Dto1D_shl_t ref, cpy2Dto1D_shl_t opt);76
bool check_cpy2Dto1D_shr_t(cpy2Dto1D_shr_t ref, cpy2Dto1D_shr_t opt);77
bool check_cpy1Dto2D_shl_t(cpy1Dto2D_shl_t ref, cpy1Dto2D_shl_t opt);78
+ bool check_cpy1Dto2D_shl_aligned_t(cpy1Dto2D_shl_t ref, cpy1Dto2D_shl_t opt);79
bool check_cpy1Dto2D_shr_t(cpy1Dto2D_shr_t ref, cpy1Dto2D_shr_t opt);80
bool check_copy_cnt_t(copy_cnt_t ref, copy_cnt_t opt);81
bool check_pixel_var(var_t ref, var_t opt);82
bool check_ssim_4x4x2_core(ssim_4x4x2_core_t ref, ssim_4x4x2_core_t opt);83
bool check_ssim_end(ssim_end4_t ref, ssim_end4_t opt);84
bool check_addAvg(addAvg_t, addAvg_t);85
+ bool check_addAvg_aligned(addAvg_t, addAvg_t);86
bool check_saoCuOrgE0_t(saoCuOrgE0_t ref, saoCuOrgE0_t opt);87
bool check_saoCuOrgE1_t(saoCuOrgE1_t ref, saoCuOrgE1_t opt);88
bool check_saoCuOrgE2_t(saoCuOrgE2_t ref[], saoCuOrgE2_t opt[]);89
x265_2.7.tar.gz/source/test/regression-tests.txt -> x265_2.9.tar.gz/source/test/regression-tests.txt
Changed
63
1
2
BasketballDrive_1920x1080_50.y4m,--preset slower --lossless --chromaloc 3 --subme 0 --limit-tu 43
BasketballDrive_1920x1080_50.y4m,--preset slower --no-cutree --analysis-save x265_analysis.dat --analysis-reuse-level 10 --bitrate 7000 --limit-tu 0::--preset slower --no-cutree --analysis-load x265_analysis.dat --analysis-reuse-level 10 --bitrate 7000 --limit-tu 04
BasketballDrive_1920x1080_50.y4m,--preset veryslow --crf 4 --cu-lossless --pmode --limit-refs 1 --aq-mode 3 --limit-tu 35
-BasketballDrive_1920x1080_50.y4m,--preset veryslow --no-cutree --analysis-save x265_analysis.dat --bitrate 7000 --tskip-fast --limit-tu 2::--preset veryslow --no-cutree --analysis-load x265_analysis.dat --bitrate 7000 --tskip-fast --limit-tu 26
+BasketballDrive_1920x1080_50.y4m,--preset veryslow --no-cutree --analysis-save x265_analysis.dat --crf 18 --tskip-fast --limit-tu 2::--preset veryslow --no-cutree --analysis-load x265_analysis.dat --crf 18 --tskip-fast --limit-tu 27
BasketballDrive_1920x1080_50.y4m,--preset veryslow --recon-y4m-exec "ffplay -i pipe:0 -autoexit"8
Coastguard-4k.y4m,--preset ultrafast --recon-y4m-exec "ffplay -i pipe:0 -autoexit"9
Coastguard-4k.y4m,--preset superfast --tune grain --overscan=crop10
Coastguard-4k.y4m,--preset superfast --tune grain --pme --aq-strength 2 --merange 19011
-Coastguard-4k.y4m,--preset veryfast --no-cutree --analysis-save x265_analysis.dat --analysis-reuse-level 1 --bitrate 15000::--preset veryfast --no-cutree --analysis-load x265_analysis.dat --analysis-reuse-level 1 --bitrate 1500012
+Coastguard-4k.y4m,--preset veryfast --no-cutree --analysis-save x265_analysis.dat --analysis-reuse-level 1 --qp 35::--preset veryfast --no-cutree --analysis-load x265_analysis.dat --analysis-reuse-level 1 --qp 3513
Coastguard-4k.y4m,--preset medium --rdoq-level 1 --tune ssim --no-signhide --me umh --slices 214
Coastguard-4k.y4m,--preset slow --tune psnr --cbqpoffs -1 --crqpoffs 1 --limit-refs 115
CrowdRun_1920x1080_50_10bit_422.yuv,--preset ultrafast --weightp --tune zerolatency --qg-size 1616
17
KristenAndSara_1280x720_60.y4m,--preset slower --pmode --max-tu-size 8 --limit-refs 0 --limit-modes --limit-tu 118
NebutaFestival_2560x1600_60_10bit_crop.yuv,--preset superfast --tune psnr19
NebutaFestival_2560x1600_60_10bit_crop.yuv,--preset medium --tune grain --limit-refs 220
-NebutaFestival_2560x1600_60_10bit_crop.yuv,--preset slow --no-cutree --analysis-save x265_analysis.dat --rd 5 --analysis-reuse-level 10 --bitrate 9000::--preset slow --no-cutree --analysis-load x265_analysis.dat --rd 5 --analysis-reuse-level 10 --bitrate 900021
+NebutaFestival_2560x1600_60_10bit_crop.yuv,--preset slow --no-cutree --analysis-save x265_analysis.dat --rd 5 --analysis-reuse-level 10 --bitrate 9000 --vbv-maxrate 9000 --vbv-bufsize 9000::--preset slow --no-cutree --analysis-load x265_analysis.dat --rd 5 --analysis-reuse-level 10 --bitrate 9000 --vbv-maxrate 9000 --vbv-bufsize 900022
News-4k.y4m,--preset ultrafast --no-cutree --analysis-save x265_analysis.dat --analysis-reuse-level 2 --bitrate 15000::--preset ultrafast --no-cutree --analysis-load x265_analysis.dat --analysis-reuse-level 2 --bitrate 1500023
News-4k.y4m,--preset superfast --lookahead-slices 6 --aq-mode 024
News-4k.y4m,--preset superfast --slices 4 --aq-mode 0 25
News-4k.y4m,--preset medium --tune ssim --no-sao --qg-size 1626
-News-4k.y4m,--preset slower --opt-cu-delta-qp27
News-4k.y4m,--preset veryslow --no-rskip28
News-4k.y4m,--preset veryslow --pme --crf 4029
OldTownCross_1920x1080_50_10bit_422.yuv,--preset superfast --weightp30
31
city_4cif_60fps.y4m,--preset superfast --rdpenalty 1 --tu-intra-depth 232
city_4cif_60fps.y4m,--preset medium --crf 4 --cu-lossless --sao-non-deblock33
city_4cif_60fps.y4m,--preset slower --scaling-list default34
-city_4cif_60fps.y4m,--preset veryslow --opt-cu-delta-qp35
city_4cif_60fps.y4m,--preset veryslow --rdpenalty 2 --sao-non-deblock --no-b-intra --limit-refs 036
ducks_take_off_420_720p50.y4m,--preset ultrafast --constrained-intra --rd 137
ducks_take_off_444_720p50.y4m,--preset superfast --weightp --limit-refs 238
39
Kimono1_1920x1080_24_400.yuv,--preset veryslow --crf 4 --cu-lossless --slices 2 --limit-refs 3 --limit-modes40
Kimono1_1920x1080_24_400.yuv,--preset placebo --ctu 32 --max-tu-size 8 --limit-tu 241
big_buck_bunny_360p24.y4m, --keyint 60 --min-keyint 40 --gop-lookahead 1442
-BasketballDrive_1920x1080_50.y4m, --preset medium --no-open-gop --keyint 50 --min-keyint 50 --radl 243
+BasketballDrive_1920x1080_50.y4m, --preset medium --no-open-gop --keyint 50 --min-keyint 50 --radl 2 --vbv-maxrate 5000 --vbv-bufsize 500044
45
# Main12 intraCost overflow bug test46
720p50_parkrun_ter.y4m,--preset medium47
48
#low-pass dct test49
720p50_parkrun_ter.y4m,--preset medium --lowpass-dct50
51
+#scaled save/load test52
+crowd_run_1080p50.y4m,--preset ultrafast --no-cutree --analysis-save x265_analysis.dat --analysis-reuse-level 1 --scale-factor 2 --crf 26 --vbv-maxrate 8000 --vbv-bufsize 8000::crowd_run_2160p50.y4m, --preset ultrafast --no-cutree --analysis-load x265_analysis.dat --analysis-reuse-level 1 --scale-factor 2 --crf 26 --vbv-maxrate 12000 --vbv-bufsize 12000 53
+crowd_run_1080p50.y4m,--preset superfast --no-cutree --analysis-save x265_analysis.dat --analysis-reuse-level 2 --scale-factor 2 --crf 22 --vbv-maxrate 5000 --vbv-bufsize 5000::crowd_run_2160p50.y4m, --preset superfast --no-cutree --analysis-load x265_analysis.dat --analysis-reuse-level 2 --scale-factor 2 --crf 22 --vbv-maxrate 10000 --vbv-bufsize 10000 54
+crowd_run_1080p50.y4m,--preset fast --no-cutree --analysis-save x265_analysis.dat --analysis-reuse-level 5 --scale-factor 2 --qp 18::crowd_run_2160p50.y4m, --preset fast --no-cutree --analysis-load x265_analysis.dat --analysis-reuse-level 5 --scale-factor 2 --qp 1855
+crowd_run_1080p50.y4m,--preset medium --no-cutree --analysis-save x265_analysis.dat --analysis-reuse-level 10 --scale-factor 2 --bitrate 5000 --vbv-maxrate 5000 --vbv-bufsize 5000 --early-skip --tu-inter-depth 3::crowd_run_2160p50.y4m, --preset medium --no-cutree --analysis-load x265_analysis.dat --analysis-reuse-level 10 --scale-factor 2 --bitrate 10000 --vbv-maxrate 10000 --vbv-bufsize 10000 --early-skip --tu-inter-depth 3 --refine-intra 4 --dynamic-refine::crowd_run_2160p50.y4m, --preset medium --no-cutree --analysis-load x265_analysis.dat --analysis-reuse-level 10 --scale-factor 2 --bitrate 10000 --vbv-maxrate 10000 --vbv-bufsize 10000 --early-skip --tu-inter-depth 3 --refine-intra 3 --refine-inter 356
+RaceHorses_416x240_30.y4m,--preset slow --no-cutree --ctu 16 --analysis-save x265_analysis.dat --analysis-reuse-level 10 --scale-factor 2 --crf 22 --vbv-maxrate 1000 --vbv-bufsize 1000::RaceHorses_832x480_30.y4m, --preset slow --no-cutree --ctu 32 --analysis-load x265_analysis.dat --analysis-save x265_analysis_2.dat --analysis-reuse-level 10 --scale-factor 2 --crf 16 --vbv-maxrate 4000 --vbv-bufsize 4000 --refine-intra 0 --refine-inter 1::RaceHorses_1664x960_30.y4m,--preset slow --no-cutree --ctu 64 --analysis-load x265_analysis_2.dat --analysis-reuse-level 10 --scale-factor 2 --crf 12 --vbv-maxrate 7000 --vbv-bufsize 7000 --refine-intra 2 --refine-inter 257
+ElFunete_960x540_60.yuv,--colorprim bt709 --transfer bt709 --chromaloc 2 --aud --repeat-headers --no-opt-qp-pps --no-opt-ref-list-length-pps --wpp --no-interlace --sar 1:1 --min-keyint 60 --no-open-gop --rc-lookahead 180 --bframes 5 --b-intra --ref 4 --cbqpoffs -2 --crqpoffs -2 --lookahead-threads 0 --weightb --qg-size 8 --me star --preset veryslow --frame-threads 1 --b-adapt 2 --aq-mode 3 --rd 6 --pools 15 --colormatrix bt709 --keyint 120 --high-tier --ctu 64 --tune psnr --bitrate 10000 --vbv-bufsize 30000 --vbv-maxrate 17500 --analysis-reuse-level 10 --analysis-save elfuente_960x540.dat --scale-factor 2::ElFunete_1920x1080_60.yuv,--colorprim bt709 --transfer bt709 --chromaloc 2 --aud --repeat-headers --no-opt-qp-pps --no-opt-ref-list-length-pps --wpp --no-interlace --sar 1:1 --min-keyint 60 --no-open-gop --rc-lookahead 180 --bframes 5 --b-intra --ref 4 --cbqpoffs -2 --crqpoffs -2 --lookahead-threads 0 --weightb --qg-size 8 --me star --preset veryslow --frame-threads 1 --b-adapt 2 --aq-mode 3 --rd 6 --pools 15 --colormatrix bt709 --keyint 120 --high-tier --ctu 64 --tune psnr --bitrate 10000 --vbv-bufsize 30000 --vbv-maxrate 17500 --analysis-reuse-level 10 --analysis-save elfuente_1920x1080.dat --limit-tu 0 --scale-factor 2 --analysis-load elfuente_960x540.dat --refine-intra 4 --refine-inter 2::ElFuente_3840x2160_60.yuv,--colorprim bt709 --transfer bt709 --chromaloc 2 --aud --repeat-headers --no-opt-qp-pps --no-opt-ref-list-length-pps --wpp --no-interlace --sar 1:1 --min-keyint 60 --no-open-gop --rc-lookahead 180 --bframes 5 --b-intra --ref 4 --cbqpoffs -2 --crqpoffs -2 --lookahead-threads 0 --weightb --qg-size 8 --me star --preset veryslow --frame-threads 1 --b-adapt 2 --aq-mode 3 --rd 6 --pools 15 --colormatrix bt709 --keyint 120 --high-tier --ctu 64 --tune=psnr --bitrate 24000 --vbv-bufsize 84000 --vbv-maxrate 49000 --analysis-reuse-level 10 --limit-tu 0 --scale-factor 2 --analysis-load elfuente_1920x1080.dat --refine-intra 4 --refine-inter 258
+59
+#segment encoding60
+BasketballDrive_1920x1080_50.y4m, --preset ultrafast --no-open-gop --chunk-start 100 --chunk-end 20061
+62
# vim: tw=20063
x265_2.7.tar.gz/source/test/smoke-tests.txt -> x265_2.9.tar.gz/source/test/smoke-tests.txt
Changed
10
1
2
old_town_cross_444_720p50.y4m,--preset=fast --keyint 20 --min-cu-size 163
old_town_cross_444_720p50.y4m,--preset=slow --sao-non-deblock --pmode --qg-size 324
RaceHorses_416x240_30_10bit.yuv,--preset=veryfast --max-tu-size 85
-RaceHorses_416x240_30_10bit.yuv,--preset=slower --bitrate 500 -F4 --rdoq-level 1 --opt-cu-delta-qp6
+RaceHorses_416x240_30_10bit.yuv,--preset=slower --bitrate 500 -F4 --rdoq-level 17
CrowdRun_1920x1080_50_10bit_444.yuv,--preset=ultrafast --constrained-intra --min-keyint 5 --keyint 108
CrowdRun_1920x1080_50_10bit_444.yuv,--preset=medium --max-tu-size 16 --tu-inter-depth 2 --limit-tu 39
DucksAndLegs_1920x1080_60_10bit_422.yuv,--preset=veryfast --min-cu 1610
x265_2.7.tar.gz/source/test/testbench.cpp -> x265_2.9.tar.gz/source/test/testbench.cpp
Changed
28
1
2
3
int main(int argc, char *argv[])4
{5
- int cpuid = X265_NS::cpu_detect();6
+ bool enableavx512 = true;7
+ int cpuid = X265_NS::cpu_detect(enableavx512);8
const char *testname = 0;9
10
if (!(argc & 1))11
12
if (!strncmp(name, "cpuid", strlen(name)))13
{14
bool bError = false;15
- cpuid = parseCpuName(value, bError);16
+ cpuid = parseCpuName(value, bError, enableavx512);17
if (bError)18
{19
printf("Invalid CPU name: %s\n", value);20
21
{ "XOP", X265_CPU_XOP },22
{ "AVX2", X265_CPU_AVX2 },23
{ "BMI2", X265_CPU_AVX2 | X265_CPU_BMI1 | X265_CPU_BMI2 },24
+ { "AVX512", X265_CPU_AVX512 },25
{ "ARMv6", X265_CPU_ARMV6 },26
{ "NEON", X265_CPU_NEON },27
{ "FastNeonMRC", X265_CPU_FAST_NEON_MRC },28
x265_2.7.tar.gz/source/test/testharness.h -> x265_2.9.tar.gz/source/test/testharness.h
Changed
19
1
2
#include <x86intrin.h>3
#elif ( !defined(__APPLE__) && defined (__GNUC__) && defined(__ARM_NEON__))4
#include <arm_neon.h>5
-#elif defined(__GNUC__)6
+#elif defined(__GNUC__) && (!defined(__clang__) || __clang_major__ < 4)7
/* fallback for older GCC/MinGW */8
static inline uint32_t __rdtsc(void)9
{10
11
}12
#endif // ifdef _MSC_VER13
14
-#define BENCH_RUNS 100015
+#define BENCH_RUNS 200016
17
// Adapted from checkasm.c, runs each optimized primitive four times, measures rdtsc18
// and discards invalid times. Repeats 1000 times to get a good average. Then measures19
x265_2.7.tar.gz/source/x265.cpp -> x265_2.9.tar.gz/source/x265.cpp
Changed
121
1
2
const char* reconPlayCmd;3
const x265_api* api;4
x265_param* param;5
+ x265_vmaf_data* vmafData;6
bool bProgress;7
bool bForceY4m;8
bool bDither;9
10
reconPlayCmd = NULL;11
api = NULL;12
param = NULL;13
+ vmafData = NULL;14
framesToBeEncoded = seek = 0;15
totalbytes = 0;16
bProgress = true;17
18
{19
int eta = (int)(elapsed * (framesToBeEncoded - frameNum) / ((int64_t)frameNum * 1000000));20
sprintf(buf, "x265 [%.1f%%] %d/%d frames, %.2f fps, %.2f kb/s, eta %d:%02d:%02d",21
- 100. * frameNum / framesToBeEncoded, frameNum, framesToBeEncoded, fps, bitrate,22
+ 100. * frameNum / (param->chunkEnd ? param->chunkEnd : param->totalFrames), frameNum, (param->chunkEnd ? param->chunkEnd : param->totalFrames), fps, bitrate,23
eta / 3600, (eta / 60) % 60, eta % 60);24
}25
else26
27
x265_log(NULL, X265_LOG_ERROR, "param alloc failed\n");28
return true;29
}30
+#if ENABLE_LIBVMAF31
+ vmafData = (x265_vmaf_data*)x265_malloc(sizeof(x265_vmaf_data));32
+ if(!vmafData)33
+ {34
+ x265_log(NULL, X265_LOG_ERROR, "vmaf data alloc failed\n");35
+ return true;36
+ }37
+#endif38
39
if (api->param_default_preset(param, preset, tune) < 0)40
{41
42
info.frameCount = 0;43
getParamAspectRatio(param, info.sarWidth, info.sarHeight);44
45
+46
this->input = InputFile::open(info, this->bForceY4m);47
if (!this->input || this->input->isFail())48
{49
50
if (this->framesToBeEncoded == 0 && info.frameCount > (int)seek)51
this->framesToBeEncoded = info.frameCount - seek;52
param->totalFrames = this->framesToBeEncoded;53
-54
+ 55
/* Force CFR until we have support for VFR */56
info.timebaseNum = param->fpsDenom;57
info.timebaseDenom = param->fpsNum;58
59
param->sourceWidth, param->sourceHeight, param->fpsNum, param->fpsDenom,60
x265_source_csp_names[param->internalCsp]);61
}62
+#if ENABLE_LIBVMAF63
+ if (!reconfn)64
+ {65
+ x265_log(param, X265_LOG_ERROR, "recon file must be specified to get VMAF score, try --help for help\n");66
+ return true;67
+ }68
+ const char *str = strrchr(info.filename, '.');69
70
+ if (!strcmp(str, ".y4m"))71
+ {72
+ x265_log(param, X265_LOG_ERROR, "VMAF supports YUV file format only.\n");73
+ return true; 74
+ }75
+ if(param->internalCsp == X265_CSP_I420 || param->internalCsp == X265_CSP_I422 || param->internalCsp == X265_CSP_I444)76
+ {77
+ vmafData->reference_file = x265_fopen(inputfn, "rb");78
+ vmafData->distorted_file = x265_fopen(reconfn, "rb");79
+ }80
+ else81
+ {82
+ x265_log(param, X265_LOG_ERROR, "VMAF will support only yuv420p, yu422p, yu444p, yuv420p10le, yuv422p10le, yuv444p10le formats.\n");83
+ return true;84
+ }85
+#endif86
this->output = OutputFile::open(outputfn, info);87
if (this->output->isFail())88
{89
90
91
x265_param* param = cliopt.param;92
const x265_api* api = cliopt.api;93
-94
+#if ENABLE_LIBVMAF95
+ x265_vmaf_data* vmafdata = cliopt.vmafData;96
+#endif97
/* This allows muxers to modify bitstream format */98
cliopt.output->setParam(param);99
100
101
if (!numEncoded)102
break;103
}104
-105
+ 106
/* clear progress report */107
if (cliopt.bProgress)108
fprintf(stderr, "%*s\r", 80, " ");109
110
111
api->encoder_get_stats(encoder, &stats, sizeof(stats));112
if (param->csvfn && !b_ctrl_c)113
+#if ENABLE_LIBVMAF114
+ api->vmaf_encoder_log(encoder, argc, argv, param, vmafdata);115
+#else116
api->encoder_log(encoder, argc, argv);117
+#endif118
api->encoder_close(encoder);119
120
int64_t second_largest_pts = 0;121
x265_2.7.tar.gz/source/x265.h -> x265_2.9.tar.gz/source/x265.h
Changed
425
1
2
extern "C" {3
#endif4
5
+#if _MSC_VER6
+#pragma warning(disable: 4201) // non-standard extension used (nameless struct/union)7
+#endif8
+9
/* x265_encoder:10
* opaque handler for encoder */11
typedef struct x265_encoder x265_encoder;12
13
int lastMiniGopBFrame;14
int plannedType[X265_LOOKAHEAD_MAX + 1];15
int64_t dts;16
+ int64_t reorderedPts;17
} x265_lookahead_data;18
19
+typedef struct x265_analysis_validate20
+{21
+ int maxNumReferences;22
+ int analysisReuseLevel;23
+ int sourceWidth;24
+ int sourceHeight;25
+ int keyframeMax;26
+ int keyframeMin;27
+ int openGOP;28
+ int bframes;29
+ int bPyramid;30
+ int maxCUSize;31
+ int minCUSize;32
+ int intraRefresh;33
+ int lookaheadDepth;34
+ int chunkStart;35
+ int chunkEnd;36
+}x265_analysis_validate;37
+38
+/* Stores intra analysis data for a single frame. This struct needs better packing */39
+typedef struct x265_analysis_intra_data40
+{41
+ uint8_t* depth;42
+ uint8_t* modes;43
+ char* partSizes;44
+ uint8_t* chromaModes;45
+}x265_analysis_intra_data;46
+47
+typedef struct x265_analysis_MV48
+{49
+ union{50
+ struct { int16_t x, y; };51
+52
+ int32_t word;53
+ };54
+}x265_analysis_MV;55
+56
+/* Stores inter analysis data for a single frame */57
+typedef struct x265_analysis_inter_data58
+{59
+ int32_t* ref;60
+ uint8_t* depth;61
+ uint8_t* modes;62
+ uint8_t* partSize;63
+ uint8_t* mergeFlag;64
+ uint8_t* interDir;65
+ uint8_t* mvpIdx[2];66
+ int8_t* refIdx[2];67
+ x265_analysis_MV* mv[2];68
+ int64_t* sadCost;69
+}x265_analysis_inter_data;70
+71
+typedef struct x265_weight_param72
+{73
+ uint32_t log2WeightDenom;74
+ int inputWeight;75
+ int inputOffset;76
+ int wtPresent;77
+}x265_weight_param;78
+79
+#if X265_DEPTH < 1080
+typedef uint32_t sse_t;81
+#else82
+typedef uint64_t sse_t;83
+#endif84
+85
+typedef struct x265_analysis_distortion_data86
+{87
+ sse_t* distortion;88
+ sse_t* ctuDistortion;89
+ double* scaledDistortion;90
+ double averageDistortion;91
+ double sdDistortion;92
+ uint32_t highDistortionCtuCount;93
+ uint32_t lowDistortionCtuCount;94
+ double* offset;95
+ double* threshold;96
+}x265_analysis_distortion_data;97
+98
/* Stores all analysis data for a single frame */99
typedef struct x265_analysis_data100
{101
- int64_t satdCost;102
- uint32_t frameRecordSize;103
- uint32_t poc;104
- uint32_t sliceType;105
- uint32_t numCUsInFrame;106
- uint32_t numPartitions;107
- uint32_t depthBytes;108
- int bScenecut;109
- void* wt;110
- void* interData;111
- void* intraData;112
- uint32_t numCuInHeight;113
- x265_lookahead_data lookahead;114
- uint8_t* modeFlag[2];115
+ int64_t satdCost;116
+ uint32_t frameRecordSize;117
+ uint32_t poc;118
+ uint32_t sliceType;119
+ uint32_t numCUsInFrame;120
+ uint32_t numPartitions;121
+ uint32_t depthBytes;122
+ int bScenecut;123
+ x265_weight_param* wt;124
+ x265_analysis_inter_data* interData;125
+ x265_analysis_intra_data* intraData;126
+ uint32_t numCuInHeight;127
+ x265_lookahead_data lookahead;128
+ uint8_t* modeFlag[2];129
+ x265_analysis_validate saveParam;130
+ x265_analysis_distortion_data* distortionData;131
} x265_analysis_data;132
133
/* cu statistics */134
135
/* All the above values will add up to 100%. */136
} x265_pu_stats;137
138
-139
-typedef struct x265_analysis_2Pass140
-{141
- uint32_t poc;142
- uint32_t frameRecordSize;143
- void* analysisFramedata;144
-}x265_analysis_2Pass;145
-146
/* Frame level statistics */147
typedef struct x265_frame_stats148
{149
150
x265_cu_stats cuStats;151
x265_pu_stats puStats;152
double totalFrameTime;153
+ double vmafFrameScore;154
+ double bufferFillFinal;155
} x265_frame_stats;156
157
typedef struct x265_ctu_info_t158
159
REGION_REFRESH_INFO = 134,160
MASTERING_DISPLAY_INFO = 137,161
CONTENT_LIGHT_LEVEL_INFO = 144,162
+ ALTERNATIVE_TRANSFER_CHARACTERISTICS = 147,163
} SEIPayloadType;164
165
typedef struct x265_sei_payload166
167
168
int height;169
170
- x265_analysis_2Pass analysis2Pass;171
+ // pts is reordered in the order of encoding.172
+ int64_t reorderedPts;173
} x265_picture;174
175
typedef enum176
177
/* CPU flags */178
179
/* x86 */180
-#define X265_CPU_CMOV 0x0000001181
-#define X265_CPU_MMX 0x0000002182
-#define X265_CPU_MMX2 0x0000004 /* MMX2 aka MMXEXT aka ISSE */183
+#define X265_CPU_MMX (1 << 0)184
+#define X265_CPU_MMX2 (1 << 1) /* MMX2 aka MMXEXT aka ISSE */185
#define X265_CPU_MMXEXT X265_CPU_MMX2186
-#define X265_CPU_SSE 0x0000008187
-#define X265_CPU_SSE2 0x0000010188
-#define X265_CPU_SSE3 0x0000020189
-#define X265_CPU_SSSE3 0x0000040190
-#define X265_CPU_SSE4 0x0000080 /* SSE4.1 */191
-#define X265_CPU_SSE42 0x0000100 /* SSE4.2 */192
-#define X265_CPU_LZCNT 0x0000200 /* Phenom support for "leading zero count" instruction. */193
-#define X265_CPU_AVX 0x0000400 /* AVX support: requires OS support even if YMM registers aren't used. */194
-#define X265_CPU_XOP 0x0000800 /* AMD XOP */195
-#define X265_CPU_FMA4 0x0001000 /* AMD FMA4 */196
-#define X265_CPU_AVX2 0x0002000 /* AVX2 */197
-#define X265_CPU_FMA3 0x0004000 /* Intel FMA3 */198
-#define X265_CPU_BMI1 0x0008000 /* BMI1 */199
-#define X265_CPU_BMI2 0x0010000 /* BMI2 */200
+#define X265_CPU_SSE (1 << 2)201
+#define X265_CPU_SSE2 (1 << 3)202
+#define X265_CPU_LZCNT (1 << 4)203
+#define X265_CPU_SSE3 (1 << 5)204
+#define X265_CPU_SSSE3 (1 << 6)205
+#define X265_CPU_SSE4 (1 << 7) /* SSE4.1 */206
+#define X265_CPU_SSE42 (1 << 8) /* SSE4.2 */207
+#define X265_CPU_AVX (1 << 9) /* Requires OS support even if YMM registers aren't used. */208
+#define X265_CPU_XOP (1 << 10) /* AMD XOP */209
+#define X265_CPU_FMA4 (1 << 11) /* AMD FMA4 */210
+#define X265_CPU_FMA3 (1 << 12) /* Intel FMA3 */211
+#define X265_CPU_BMI1 (1 << 13) /* BMI1 */212
+#define X265_CPU_BMI2 (1 << 14) /* BMI2 */213
+#define X265_CPU_AVX2 (1 << 15) /* AVX2 */214
+#define X265_CPU_AVX512 (1 << 16) /* AVX-512 {F, CD, BW, DQ, VL}, requires OS support */215
/* x86 modifiers */216
-#define X265_CPU_CACHELINE_32 0x0020000 /* avoid memory loads that span the border between two cachelines */217
-#define X265_CPU_CACHELINE_64 0x0040000 /* 32/64 is the size of a cacheline in bytes */218
-#define X265_CPU_SSE2_IS_SLOW 0x0080000 /* avoid most SSE2 functions on Athlon64 */219
-#define X265_CPU_SSE2_IS_FAST 0x0100000 /* a few functions are only faster on Core2 and Phenom */220
-#define X265_CPU_SLOW_SHUFFLE 0x0200000 /* The Conroe has a slow shuffle unit (relative to overall SSE performance) */221
-#define X265_CPU_STACK_MOD4 0x0400000 /* if stack is only mod4 and not mod16 */222
-#define X265_CPU_SLOW_CTZ 0x0800000 /* BSR/BSF x86 instructions are really slow on some CPUs */223
-#define X265_CPU_SLOW_ATOM 0x1000000 /* The Atom is terrible: slow SSE unaligned loads, slow224
+#define X265_CPU_CACHELINE_32 (1 << 17) /* avoid memory loads that span the border between two cachelines */225
+#define X265_CPU_CACHELINE_64 (1 << 18) /* 32/64 is the size of a cacheline in bytes */226
+#define X265_CPU_SSE2_IS_SLOW (1 << 19) /* avoid most SSE2 functions on Athlon64 */227
+#define X265_CPU_SSE2_IS_FAST (1 << 20) /* a few functions are only faster on Core2 and Phenom */228
+#define X265_CPU_SLOW_SHUFFLE (1 << 21) /* The Conroe has a slow shuffle unit (relative to overall SSE performance) */229
+#define X265_CPU_STACK_MOD4 (1 << 22) /* if stack is only mod4 and not mod16 */230
+#define X265_CPU_SLOW_ATOM (1 << 23) /* The Atom is terrible: slow SSE unaligned loads, slow231
* SIMD multiplies, slow SIMD variable shifts, slow pshufb,232
* cacheline split penalties -- gather everything here that233
* isn't shared by other CPUs to avoid making half a dozen234
* new SLOW flags. */235
-#define X265_CPU_SLOW_PSHUFB 0x2000000 /* such as on the Intel Atom */236
-#define X265_CPU_SLOW_PALIGNR 0x4000000 /* such as on the AMD Bobcat */237
+#define X265_CPU_SLOW_PSHUFB (1 << 24) /* such as on the Intel Atom */238
+#define X265_CPU_SLOW_PALIGNR (1 << 25) /* such as on the AMD Bobcat */239
240
/* ARM */241
#define X265_CPU_ARMV6 0x0000001242
243
#define X265_AQ_VARIANCE 1244
#define X265_AQ_AUTO_VARIANCE 2245
#define X265_AQ_AUTO_VARIANCE_BIASED 3246
-247
#define x265_ADAPT_RD_STRENGTH 4248
-249
+#define X265_REFINE_INTER_LEVELS 3250
/* NOTE! For this release only X265_CSP_I420 and X265_CSP_I444 are supported */251
-252
/* Supported internal color space types (according to semantics of chroma_format_idc) */253
#define X265_CSP_I400 0 /* yuv 4:0:0 planar */254
#define X265_CSP_I420 1 /* yuv 4:2:0 planar */255
256
double elapsedEncodeTime; /* wall time since encoder was opened */257
double elapsedVideoTime; /* encoded picture count / frame rate */258
double bitrate; /* accBits / elapsed video time */259
+ double aggregateVmafScore; /* aggregate VMAF score for input video*/260
uint64_t accBits; /* total bits output thus far */261
uint32_t encodedPictureCount; /* number of output pictures thus far */262
uint32_t totalWPFrames; /* number of uni-directional weighted frames used */263
264
float bitrateFactor;265
} x265_zone;266
267
+/* data to calculate aggregate VMAF score */268
+typedef struct x265_vmaf_data269
+{270
+ int width;271
+ int height;272
+ size_t offset; 273
+ int internalBitDepth;274
+ FILE *reference_file; /* FILE pointer for input file */275
+ FILE *distorted_file; /* FILE pointer for recon file generated*/276
+}x265_vmaf_data;277
+278
+/* data to calculate frame level VMAF score */279
+typedef struct x265_vmaf_framedata280
+{281
+ int width;282
+ int height;283
+ int frame_set; 284
+ int internalBitDepth; 285
+ void *reference_frame; /* points to fenc of particular frame */286
+ void *distorted_frame; /* points to recon of particular frame */287
+}x265_vmaf_framedata;288
+289
+/* common data needed to calculate both frame level and video level VMAF scores */290
+typedef struct x265_vmaf_commondata291
+{292
+ char *format;293
+ char *model_path;294
+ char *log_path;295
+ char *log_fmt;296
+ int disable_clip;297
+ int disable_avx;298
+ int enable_transform;299
+ int phone_model;300
+ int psnr;301
+ int ssim;302
+ int ms_ssim;303
+ char *pool;304
+}x265_vmaf_commondata;305
+306
+static const x265_vmaf_commondata vcd[] = { { NULL, (char *)"/usr/local/share/model/vmaf_v0.6.1.pkl", NULL, NULL, 0, 0, 0, 0, 0, 0, 0, NULL } };307
+308
/* x265 input parameters309
*310
* For version safety you may use x265_param_alloc/free() to manage the311
312
* somehow flawed on your target hardware. The asm function tables are313
* process global, the first encoder configures them for all encoders */314
int cpuid;315
-316
/*== Parallelism Features ==*/317
318
/* Number of concurrently encoded frames between 1 and X265_MAX_FRAME_THREADS319
320
* Default is 0, which is recommended */321
int crQpOffset;322
323
+ /* Specifies the preferred transfer characteristics syntax element in the324
+ * alternative transfer characteristics SEI message (see. D.2.38 and D.3.38 of325
+ * JCTVC-W1005 http://phenix.it-sudparis.eu/jct/doc_end_user/documents/23_San%20Diego/wg11/JCTVC-W1005-v4.zip326
+ * */327
+ int preferredTransferCharacteristics;328
+ 329
+ /*330
+ * Specifies the value for the pic_struc syntax element of the picture timing SEI message (See D2.3 and D3.3)331
+ * of the HEVC spec. for a detailed explanation332
+ * */333
+ int pictureStructure; 334
+335
struct336
{337
/* Explicit mode of rate-control, necessary for API users. It must338
339
340
/*Number of RADL pictures allowed in front of IDR*/341
int radl;342
+343
+ /* This value controls the maximum AU size defined in specification344
+ * It represents the percentage of maximum AU size used.345
+ * Default is 1 (which is 100%). Range is 0.5 to 1. */346
+ double maxAUSizeFactor;347
+348
+ /* Enables the emission of a Recovery Point SEI with the stream headers349
+ * at each IDR frame describing poc of the recovery point, exact matching flag350
+ * and broken link flag. Default is disabled. */351
+ int bEmitIDRRecoverySEI;352
+353
+ /* Dynamically change refine-inter at block level*/354
+ int bDynamicRefine;355
+356
+ /* Enable writing all SEI messgaes in one single NAL instead of mul*/357
+ int bSingleSeiNal;358
+359
+360
+ /* First frame of the chunk. Frames preceeding this in display order will361
+ * be encoded, however, they will be discarded in the bitstream.362
+ * Default 0 (disabled). */363
+ int chunkStart;364
+365
+ /* Last frame of the chunk. Frames following this in display order will be366
+ * used in taking lookahead decisions, but, they will not be encoded.367
+ * Default 0 (disabled). */368
+ int chunkEnd;369
+ /* File containing base64 encoded SEI messages in POC order */370
+ const char* naluFile;371
+372
} x265_param;373
374
/* x265_param_alloc:375
376
* A static string describing the compiler and target architecture */377
X265_API extern const char *x265_build_info_str;378
379
+/* x265_alloc_analysis_data:380
+* Allocate memory for the x265_analysis_data object's internal structures. */381
+void x265_alloc_analysis_data(x265_param *param, x265_analysis_data* analysis);382
+383
+/*384
+* Free the allocated memory for x265_analysis_data object's internal structures. */385
+void x265_free_analysis_data(x265_param *param, x265_analysis_data* analysis);386
+387
/* Force a link error in the case of linking against an incompatible API version.388
* Glue #defines exist to force correct macro expansion; the final output of the macro389
* is x265_encoder_open_##X265_BUILD (for purposes of dlopen). */390
391
/* In-place downshift from a bit-depth greater than 8 to a bit-depth of 8, using392
* the residual bits to dither each row. */393
void x265_dither_image(x265_picture *, int picWidth, int picHeight, int16_t *errorBuf, int bitDepth);394
+#if ENABLE_LIBVMAF395
+/* x265_calculate_vmafScore:396
+ * returns VMAF score for the input video.397
+ * This api must be called only after encoding was done. */398
+double x265_calculate_vmafscore(x265_param*, x265_vmaf_data*);399
+400
+/* x265_calculate_vmaf_framelevelscore:401
+ * returns VMAF score for each frame in a given input video. */402
+double x265_calculate_vmaf_framelevelscore(x265_vmaf_framedata*);403
+/* x265_vmaf_encoder_log:404
+ * write a line to the configured CSV file. If a CSV filename was not405
+ * configured, or file open failed, this function will perform no write.406
+ * This api will be called only when ENABLE_LIBVMAF cmake option is set */407
+void x265_vmaf_encoder_log(x265_encoder *encoder, int argc, char **argv, x265_param*, x265_vmaf_data*);408
+409
+#endif410
411
#define X265_MAJOR_VERSION 1412
413
414
void (*csvlog_encode)(const x265_param*, const x265_stats *, int, int, int, char**);415
void (*dither_image)(x265_picture*, int, int, int16_t*, int);416
int (*set_analysis_data)(x265_encoder *encoder, x265_analysis_data *analysis_data, int poc, uint32_t cuBytes);417
+#if ENABLE_LIBVMAF418
+ double (*calculate_vmafscore)(x265_param *, x265_vmaf_data *);419
+ double (*calculate_vmaf_framelevelscore)(x265_vmaf_framedata *);420
+ void (*vmaf_encoder_log)(x265_encoder*, int, char**, x265_param *, x265_vmaf_data *);421
+#endif422
/* add new pointers to the end, or increment X265_MAJOR_VERSION */423
} x265_api;424
425
x265_2.7.tar.gz/source/x265cli.h -> x265_2.9.tar.gz/source/x265cli.h
Changed
104
1
2
{ "vbv-init", required_argument, NULL, 0 },3
{ "vbv-end", required_argument, NULL, 0 },4
{ "vbv-end-fr-adj", required_argument, NULL, 0 },5
+ { "chunk-start", required_argument, NULL, 0 },6
+ { "chunk-end", required_argument, NULL, 0 },7
{ "bitrate", required_argument, NULL, 0 },8
{ "qp", required_argument, NULL, 'q' },9
{ "aq-mode", required_argument, NULL, 0 },10
11
{ "scale-factor", required_argument, NULL, 0 },12
{ "refine-intra", required_argument, NULL, 0 },13
{ "refine-inter", required_argument, NULL, 0 },14
+ { "dynamic-refine", no_argument, NULL, 0 },15
+ { "no-dynamic-refine", no_argument, NULL, 0 },16
{ "strict-cbr", no_argument, NULL, 0 },17
{ "temporal-layers", no_argument, NULL, 0 },18
{ "no-temporal-layers", no_argument, NULL, 0 },19
20
{ "refine-mv-type", required_argument, NULL, 0 },21
{ "copy-pic", no_argument, NULL, 0 },22
{ "no-copy-pic", no_argument, NULL, 0 },23
+ { "max-ausize-factor", required_argument, NULL, 0 },24
+ { "idr-recovery-sei", no_argument, NULL, 0 },25
+ { "no-idr-recovery-sei", no_argument, NULL, 0 },26
+ { "single-sei", no_argument, NULL, 0 },27
+ { "no-single-sei", no_argument, NULL, 0 },28
+ { "atc-sei", required_argument, NULL, 0 },29
+ { "pic-struct", required_argument, NULL, 0 },30
+ { "nalu-file", required_argument, NULL, 0 },31
{ 0, 0, 0, 0 },32
{ 0, 0, 0, 0 },33
{ 0, 0, 0, 0 },34
35
H0(" --dhdr10-info <filename> JSON file containing the Creative Intent Metadata to be encoded as Dynamic Tone Mapping\n");36
H0(" --[no-]dhdr10-opt Insert tone mapping SEI only for IDR frames and when the tone mapping information changes. Default disabled\n");37
#endif38
+ H0(" --nalu-file <filename> Text file containing SEI messages in the following format : <POC><space><PREFIX><space><NAL UNIT TYPE>/<SEI TYPE><space><SEI Payload>\n");39
H0("-f/--frames <integer> Maximum number of frames to encode. Default all\n");40
H0(" --seek <integer> First frame to encode\n");41
H1(" --[no-]interlace <bff|tff> Indicate input pictures are interlace fields in temporal order. Default progressive\n");42
43
H0(" --[no-]early-skip Enable early SKIP detection. Default %s\n", OPT(param->bEnableEarlySkip));44
H0(" --[no-]rskip Enable early exit from recursion. Default %s\n", OPT(param->bEnableRecursionSkip));45
H1(" --[no-]tskip-fast Enable fast intra transform skipping. Default %s\n", OPT(param->bEnableTSkipFast));46
- H1(" --[no-]splitrd-skip Enable skipping split RD analysis when sum of split CU rdCost larger than none split CU rdCost for Intra CU. Default %s\n", OPT(param->bEnableSplitRdSkip));47
+ H1(" --[no-]splitrd-skip Enable skipping split RD analysis when sum of split CU rdCost larger than one split CU rdCost for Intra CU. Default %s\n", OPT(param->bEnableSplitRdSkip));48
H1(" --nr-intra <integer> An integer value in range of 0 to 2000, which denotes strength of noise reduction in intra CUs. Default 0\n");49
H1(" --nr-inter <integer> An integer value in range of 0 to 2000, which denotes strength of noise reduction in inter CUs. Default 0\n");50
H0(" --ctu-info <integer> Enable receiving ctu information asynchronously and determine reaction to the CTU information (0, 1, 2, 4, 6) Default 0\n"51
52
H0(" --vbv-init <float> Initial VBV buffer occupancy (fraction of bufsize or in kbits). Default %.2f\n", param->rc.vbvBufferInit);53
H0(" --vbv-end <float> Final VBV buffer emptiness (fraction of bufsize or in kbits). Default 0 (disabled)\n");54
H0(" --vbv-end-fr-adj <float> Frame from which qp has to be adjusted to achieve final decode buffer emptiness. Default 0\n");55
+ H0(" --chunk-start <integer> First frame of the chunk. Default 0 (disabled)\n");56
+ H0(" --chunk-end <integer> Last frame of the chunk. Default 0 (disabled)\n");57
H0(" --pass Multi pass rate control.\n"58
" - 1 : First pass, creates stats file\n"59
" - 2 : Last pass, does not overwrite stats file\n"60
61
H0(" --analysis-reuse-level <1..10> Level of analysis reuse indicates amount of info stored/reused in save/load mode, 1:least..10:most. Default %d\n", param->analysisReuseLevel);62
H0(" --refine-mv-type <string> Reuse MV information received through API call. Supported option is avc. Default disabled - %d\n", param->bMVType);63
H0(" --scale-factor <int> Specify factor by which input video is scaled down for analysis save mode. Default %d\n", param->scaleFactor);64
- H0(" --refine-intra <0..3> Enable intra refinement for encode that uses analysis-load.\n"65
+ H0(" --refine-intra <0..4> Enable intra refinement for encode that uses analysis-load.\n"66
" - 0 : Forces both mode and depth from the save encode.\n"67
" - 1 : Functionality of (0) + evaluate all intra modes at min-cu-size's depth when current depth is one smaller than min-cu-size's depth.\n"68
" - 2 : Functionality of (1) + irrespective of size evaluate all angular modes when the save encode decides the best mode as angular.\n"69
" - 3 : Functionality of (1) + irrespective of size evaluate all intra modes.\n"70
+ " - 4 : Re-evaluate all intra blocks, does not reuse data from save encode.\n"71
" Default:%d\n", param->intraRefine);72
H0(" --refine-inter <0..3> Enable inter refinement for encode that uses analysis-load.\n"73
" - 0 : Forces both mode and depth from the save encode.\n"74
75
" - 2 : Functionality of (1) + irrespective of size restrict the modes evaluated when specific modes are decided as the best mode by the save encode.\n"76
" - 3 : Functionality of (1) + irrespective of size evaluate all inter modes.\n"77
" Default:%d\n", param->interRefine);78
+ H0(" --[no-]dynamic-refine Dynamically changes refine-inter level for each CU. Default %s\n", OPT(param->bDynamicRefine));79
H0(" --[no-]refine-mv Enable mv refinement for load mode. Default %s\n", OPT(param->mvRefine));80
H0(" --aq-mode <integer> Mode for Adaptive Quantization - 0:none 1:uniform AQ 2:auto variance 3:auto variance with bias to dark scenes. Default %d\n", param->rc.aqMode);81
H0(" --aq-strength <float> Reduces blocking and blurring in flat and textured areas (0 to 3.0). Default %.2f\n", param->rc.aqStrength);82
83
H1(" MAX_MAX_QP+1 floats for lambda table, then again for lambda2 table\n");84
H1(" Blank lines and lines starting with hash(#) are ignored\n");85
H1(" Comma is considered to be white-space\n");86
+ H0(" --max-ausize-factor <float> This value controls the maximum AU size defined in specification.\n");87
+ H0(" It represents the percentage of maximum AU size used. Default %.1f\n", param->maxAUSizeFactor);88
H0("\nLoop filters (deblock and SAO):\n");89
H0(" --[no-]deblock Enable Deblocking Loop Filter, optionally specify tC:Beta offsets Default %s\n", OPT(param->bEnableLoopFilter));90
H0(" --[no-]sao Enable Sample Adaptive Offset. Default %s\n", OPT(param->bEnableSAO));91
92
H0(" --[no-]repeat-headers Emit SPS and PPS headers at each keyframe. Default %s\n", OPT(param->bRepeatHeaders));93
H0(" --[no-]info Emit SEI identifying encoder and parameters. Default %s\n", OPT(param->bEmitInfoSEI));94
H0(" --[no-]hrd Enable HRD parameters signaling. Default %s\n", OPT(param->bEmitHRDSEI));95
+ H0(" --[no-]idr-recovery-sei Emit recovery point infor SEI at each IDR frame \n");96
H0(" --[no-]temporal-layers Enable a temporal sublayer for unreferenced B frames. Default %s\n", OPT(param->bEnableTemporalSubLayers));97
H0(" --[no-]aud Emit access unit delimiters at the start of each access unit. Default %s\n", OPT(param->bEnableAccessUnitDelimiters));98
H1(" --hash <integer> Decoded Picture Hash SEI 0: disabled, 1: MD5, 2: CRC, 3: Checksum. Default %d\n", param->decodedPictureHashSEI);99
+ H0(" --atc-sei <integer> Emit the alternative transfer characteristics SEI message where the integer is the preferred transfer characteristics. Default disabled\n");100
+ H0(" --pic-struct <integer> Set the picture structure and emits it in the picture timing SEI message. Values in the range 0..12. See D.3.3 of the HEVC spec. for a detailed explanation.\n");101
H0(" --log2-max-poc-lsb <integer> Maximum of the picture order count\n");102
H0(" --[no-]vui-timing-info Emit VUI timing information in the bistream. Default %s\n", OPT(param->bEmitVUITimingInfo));103
H0(" --[no-]vui-hrd-info Emit VUI HRD information in the bistream. Default %s\n", OPT(param->bEmitVUIHRDInfo));104
Refresh
No build results available
Refresh
No rpmlint results available
Login required, please
login
or
signup
in order to comment
