Optimalkan Subquery dengan Fungsi Windowing

Karena keterampilan penyesuaian kinerja saya sepertinya tidak pernah terasa cukup, saya selalu bertanya-tanya apakah ada lebih banyak optimasi yang dapat saya lakukan terhadap beberapa pertanyaan. Situasi yang terkait dengan pertanyaan ini adalah fungsi Windowed MAX yang bersarang di dalam subquery.

Data yang saya gali adalah serangkaian transaksi pada berbagai kelompok set yang lebih besar. Saya memiliki 4 bidang penting, ID unik transaksi, ID Grup dari sekelompok transaksi, dan tanggal yang terkait dengan transaksi unik atau grup transaksi masing-masing. Paling sering, Tanggal Grup cocok dengan Tanggal Transaksi Unik Maksimum untuk suatu Batch, tetapi ada kalanya penyesuaian manual datang melalui sistem kami dan operasi tanggal unik terjadi setelah tanggal transaksi grup diambil. Hasil edit manual ini tidak menyesuaikan tanggal grup berdasarkan desain.

Apa yang saya identifikasi dalam kueri ini adalah catatan di mana Tanggal Unik jatuh setelah Tanggal Grup. Kueri sampel berikut membangun setara kasar dari skenario saya dan pernyataan SELECT mengembalikan catatan yang saya cari, namun, apakah saya mendekati solusi ini dengan cara yang paling efisien? Ini memakan waktu cukup lama untuk dijalankan selama tabel fakta saya memuat karena catatan saya menghitung angka di 9 digit atas, tetapi kebanyakan penghinaan saya untuk subqueries membuat saya bertanya-tanya apakah ada pendekatan yang lebih baik di sini. Saya tidak khawatir tentang indeks apa pun karena saya yakin sudah ada di tempatnya; apa yang saya cari adalah pendekatan kueri alternatif yang akan mencapai hal yang sama, tetapi bahkan lebih efisien. Setiap umpan balik diterima.

CREATE TABLE #Example
(
    UniqueID INT IDENTITY(1,1)
  , GroupID INT
  , GroupDate DATETIME
  , UniqueDate DATETIME
)

CREATE CLUSTERED INDEX [CX_1] ON [#Example]
(
    [UniqueID] ASC
)


SET NOCOUNT ON

--Populate some test data
DECLARE @i INT = 0, @j INT = 5, @UniqueDate DATETIME, @GroupDate DATETIME

WHILE @i < 10000
BEGIN

    IF((@i + @j)%173 = 0)
    BEGIN
        SET @UniqueDate = GETDATE()+@i+5
    END
    ELSE
    BEGIN
        SET @UniqueDate = GETDATE()+@i
    END

    SET @GroupDate = GETDATE()+(@j-1)

    INSERT INTO #Example (GroupID, GroupDate, UniqueDate)
    VALUES (@j, @GroupDate, @UniqueDate)

    SET @i = @i + 1

    IF (@i % 5 = 0)
    BEGIN
        SET @j = @j+5
    END
END
SET NOCOUNT OFF

CREATE NONCLUSTERED INDEX [IX_2_4_3] ON [#Example]
(
    [GroupID] ASC,
    [UniqueDate] ASC,
    [GroupDate] ASC
)
INCLUDE ([UniqueID])

-- Identify any UniqueDates that are greater than the GroupDate within their GroupID
SELECT UniqueID
     , GroupID
     , GroupDate
     , UniqueDate
FROM (
    SELECT UniqueID
         , GroupID
         , GroupDate
         , UniqueDate
         , MAX(UniqueDate) OVER (PARTITION BY GroupID) AS maxUniqueDate
    FROM #Example
    ) calc_maxUD
WHERE maxUniqueDate > GroupDate
    AND maxUniqueDate = UniqueDate

DROP TABLE #Example

Aku di sini

Saya berasumsi tidak ada indeks, karena Anda belum memberikannya.

Langsung saja, indeks berikut ini akan menghilangkan operator Sortir dalam paket Anda, yang jika tidak akan berpotensi menghabiskan banyak memori:

CREATE INDEX IX ON #Example (GroupID, UniqueDate) INCLUDE (UniqueID, GroupDate);

Subquery bukan masalah kinerja dalam hal ini. Jika ada, saya akan mencari cara untuk menghilangkan fungsi jendela (MAX ... LEBIH) untuk menghindari konstruksi Nested Loop dan Table Spool.

Dengan indeks yang sama, permintaan berikut pada pandangan pertama mungkin terlihat kurang efisien, dan itu pergi dari dua hingga tiga pemindaian pada tabel dasar, tetapi menghilangkan sejumlah besar pembacaan secara internal karena tidak memiliki operator Spool. Saya menduga itu masih akan berkinerja lebih baik, terutama jika Anda memiliki core CPU dan kinerja IO yang cukup di server Anda:

SELECT e.UniqueID
     , e.GroupID
     , e.GroupDate
     , e.UniqueDate
FROM (
    SELECT GroupID, MAX(UniqueDate) AS maxUniqueDate
    FROM #Example
    GROUP BY GroupID) AS agg
INNER JOIN #Example AS e ON agg.GroupID=e.GroupID
WHERE agg.maxUniqueDate > e.GroupDate
    AND agg.maxUniqueDate = e.UniqueDate
OPTION (MERGE JOIN);

(Catatan: Saya menambahkan MERGE JOINpetunjuk permintaan, tetapi ini mungkin akan terjadi secara otomatis jika statistik Anda sesuai. Praktik terbaik adalah meninggalkan petunjuk seperti ini jika Anda bisa.)

Kapan dan jika Anda dapat memutakhirkan dari SQL Server 2012 ke SQL Server 2016, Anda mungkin dapat memanfaatkan kinerja yang jauh lebih baik (terutama untuk agregat jendela tanpa bingkai) yang disediakan oleh mode batch baru Window Aggregate operator.

Hampir semua skenario pemrosesan data besar bekerja lebih baik dengan penyimpanan columnstore daripada rowstore. Bahkan tanpa mengubah ke columnstore untuk tabel dasar Anda, Anda masih dapat memperoleh manfaat dari operator baru 2016 dan eksekusi mode batch dengan membuat indeks yang disaring kolomstore kosong yang tidak tercakup di salah satu tabel dasar, atau dengan bergabung secara berlebihan di luar ke suatu columnstore yang diorganisir meja.

Menggunakan opsi kedua, kueri menjadi:

-- Just to get batch mode processing and the window aggregate operator
CREATE TABLE #Dummy (a integer NOT NULL, INDEX DummyCC CLUSTERED COLUMNSTORE);

-- Identify any UniqueDates that are greater than the GroupDate within their GroupID
SELECT
    calc_maxUD.UniqueID,
    calc_maxUD.GroupID,
    calc_maxUD.GroupDate,
    calc_maxUD.UniqueDate
FROM 
(
    SELECT
        E.UniqueID,
        E.GroupID,
        E.GroupDate,
        E.UniqueDate,
        maxUniqueDate = MAX(UniqueDate) OVER (
            PARTITION BY GroupID)
    FROM #Example AS E
    LEFT JOIN #Dummy AS D -- The only change to the original query
        ON 1 = 0
) AS calc_maxUD
WHERE 
    calc_maxUD.maxUniqueDate > calc_maxUD.GroupDate
    AND calc_maxUD.maxUniqueDate = calc_maxUD.UniqueDate;

db <> biola

Perhatikan satu-satunya perubahan ke kueri asli adalah membuat tabel sementara yang kosong dan menambahkan gabungan kiri. Rencana pelaksanaannya adalah:

(58 row(s) affected)
Table 'Worktable'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0
Table '#Example'. Scan count 1, logical reads 40, physical reads 0, read-ahead reads 0

Untuk informasi dan opsi lebih lanjut, lihat seri Itzik Ben-Gan yang hebat, Apa yang Perlu Anda Ketahui tentang Agregat Jendela Batch Mode di SQL Server 2016 (dalam tiga bagian).

Aku hanya akan membuang ol 'Cross. Melamar di luar sana:

SELECT e.*
    FROM #Example AS e
    CROSS APPLY ( SELECT TOP 1 e2.UniqueDate AS maxUniqueDate
                    FROM #Example AS e2
                    WHERE e2.GroupID = e.GroupID 
                    ORDER BY e2.UniqueDate DESC
                    ) AS ca
    WHERE ca.maxUniqueDate > e.GroupDate
        AND ca.maxUniqueDate = e.UniqueDate;

Dengan beberapa indeks apa pun, itu cukup baik.

CREATE CLUSTERED INDEX cx_whatever ON #Example (GroupID)

CREATE UNIQUE NONCLUSTERED INDEX ix_whatever ON #Example (GroupID, UniqueDate DESC, GroupDate)

Waktu statistik dan io terlihat seperti ini (permintaan Anda adalah hasil pertama)

Table 'Worktable'. Scan count 3, logical reads 28004, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table '#Example'. Scan count 1, logical reads 51, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.

 SQL Server Execution Times:
   CPU time = 15 ms,  elapsed time = 20 ms.

Table '#Example'. Scan count 10001, logical reads 21336, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.

 SQL Server Execution Times:
   CPU time = 16 ms,  elapsed time = 11 ms.

Paket kueri ada di sini (lagi, milikmu adalah yang pertama):

https://www.brentozar.com/pastetheplan/?id=BJYJvqAal

Kenapa saya lebih suka versi ini? Saya menghindari gulungan. Jika mereka mulai tumpah ke disk, itu akan menjadi jelek.

Tetapi Anda mungkin ingin mencobanya juga.

SELECT e.*
    FROM #Example AS e
    CROSS APPLY ( SELECT e2.UniqueDate AS maxUniqueDate
                    FROM #Example AS e2
                    WHERE e2.GroupID = e.GroupID 
                    ) AS ca
    WHERE ca.maxUniqueDate > e.GroupDate
        AND ca.maxUniqueDate = e.UniqueDate;

Jika ini adalah DW besar, Anda mungkin lebih suka Hash Join, dan pemfilteran baris dalam join, daripada di akhir dalam TOP 1kueri sebagai operator Filter.

Paket ada di sini: https://www.brentozar.com/pastetheplan/?id=BkUF55ATx

Waktu statistik dan io di sini:

Table 'Workfile'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'Worktable'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table '#Example'. Scan count 2, logical reads 84, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.

 SQL Server Execution Times:
   CPU time = 16 ms,  elapsed time = 5 ms.

Semoga ini membantu!

Satu edit, berdasarkan ide @ ypercube, dan indeks baru.

CREATE NONCLUSTERED INDEX ix_meh ON #Example (UniqueDate,GroupDate) INCLUDE (UniqueID,GroupID);

WITH t1 AS 
(
    SELECT DISTINCT
    e.GroupID ,
    MAX(UniqueDate) AS MaxUniqueDate
    FROM #Example AS e
    GROUP BY e.GroupID
)
SELECT *
FROM #Example AS e
CROSS APPLY (
SELECT *
FROM t1
    WHERE t1.MaxUniqueDate > e.GroupDate
        AND t1.MaxUniqueDate = e.UniqueDate
        AND t1.GroupID = e.GroupID
) ca

Inilah statistik waktu dan io:

Table 'Workfile'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'Worktable'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table '#Example'. Scan count 2, logical reads 91, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.

 SQL Server Execution Times:
   CPU time = 0 ms,  elapsed time = 4 ms.

Inilah rencananya:

https://www.brentozar.com/pastetheplan/?id=SJv8foR6g

Saya akan melihat top with ties

Jika GroupDatesama per GroupIdmaka:

select top 1 with ties 
   UniqueID
 , GroupID
 , GroupDate
 , UniqueDate
from #Example
where UniqueDate > GroupDate
order by row_number() over (partition by GroupId order by UniqueDate desc)

Lain: gunakan top with tiesdalam ekspresi tabel umum

with cte as (
  select top 1 with ties 
      UniqueID
    , GroupID
    , GroupDate
    , UniqueDate
  from #Example
  order by row_number() over (partition by GroupId order by UniqueDate desc)
)
select *
from cte
where UniqueDate > GroupDate

dbfiddle: http://dbfiddle.uk/?rdbms=sqlserver_2016&fiddle=c058994c2f5f3d99b212f06e1dae9fd3

Permintaan Asli

Table 'Worktable'. Scan count 3, logical reads 28001, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table '#Example____________________________________________________________________________________________________________0000000000CB'. Scan count 1, logical reads 43, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.

 SQL Server Execution Times:
   CPU time = 31 ms,  elapsed time = 31 ms.

vs top with tiesdalam ekspresi tabel umum

Table 'Worktable'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table '#Example____________________________________________________________________________________________________________0000000000CB'. Scan count 1, logical reads 43, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.

 SQL Server Execution Times:
   CPU time = 16 ms,  elapsed time = 15 ms.

Jadi saya melakukan beberapa analisis pada berbagai pendekatan yang diposting sejauh ini, dan di lingkungan saya, sepertinya pendekatan Daniel menang secara konsisten pada waktu eksekusi. Anehnya (bagi saya) pendekatan CROSS APPLY ketiga sp_BlitzErik tidak jauh di belakang. Di sini ada keluaran jika ada yang tertarik, tapi terima kasih banyak untuk semua pendekatan alternatif. Saya belajar lebih banyak dari menggali jawaban atas pertanyaan ini daripada yang saya miliki cukup lama!

Windowed Function - baseline metric

(10406 row(s) affected)
Table 'DateDim'. Scan count 9, logical reads 791, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'TableFact'. Scan count 9, logical reads 140181, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'Workfile'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'Worktable'. Scan count 89815, logical reads 42553550, physical reads 0, read-ahead reads 84586, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'Table01Dim'. Scan count 9, logical reads 7688, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'Table02Dim'. Scan count 9, logical reads 7819, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.

 SQL Server Execution Times:
   CPU time = 87753 ms,  elapsed time = 13031 ms.
Warning: Null value is eliminated by an aggregate or other SET operation.


Basic Aggregated Subquery - Daniel Hutmacher

(10406 row(s) affected)
Table 'DateDim'. Scan count 18, logical reads 1194, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'TableFact'. Scan count 18, logical reads 280362, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'Workfile'. Scan count 48, logical reads 82408, physical reads 9629, read-ahead reads 72779, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'Worktable'. Scan count 89791, logical reads 6861425, physical reads 0, read-ahead reads 14565, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'Table01Dim'. Scan count 9, logical reads 7688, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'Table02Dim'. Scan count 18, logical reads 15726, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.

 SQL Server Execution Times:
   CPU time = 40527 ms,  elapsed time = 6182 ms.
Warning: Null value is eliminated by an aggregate or other SET operation.


CROSS APPLY Operation A - sp_BlitzErik

(10406 row(s) affected)
Table 'DateDim'. Scan count 9, logical reads 6199331, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'TableFact'. Scan count 3099273, logical reads 12844012, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'Table01Dim'. Scan count 3109676, logical reads 9350502, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'Worktable'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'Table02Dim'. Scan count 3109676, logical reads 9482456, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'Workfile'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'Worktable'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'Worktable'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.

 SQL Server Execution Times:
   CPU time = 132632 ms,  elapsed time = 20955 ms.


CROSS APPLY Operation C - sp_BlitzErik

(10406 row(s) affected)
Table 'DateDim'. Scan count 18, logical reads 1194, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'TableFact'. Scan count 18, logical reads 280362, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'Workfile'. Scan count 56, logical reads 92800, physical reads 10872, read-ahead reads 81928, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'Worktable'. Scan count 89791, logical reads 6861425, physical reads 0, read-ahead reads 14563, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'Table01Dim'. Scan count 18, logical reads 15376, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'Table02Dim'. Scan count 18, logical reads 15726, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.

 SQL Server Execution Times:
   CPU time = 46082 ms,  elapsed time = 6804 ms.
Warning: Null value is eliminated by an aggregate or other SET operation.


TOP 1 WITH TIES - B - SqlZim

(10406 row(s) affected)
Table 'DateDim'. Scan count 9, logical reads 791, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'TableFact'. Scan count 9, logical reads 140181, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'Workfile'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'Worktable'. Scan count 89791, logical reads 6866304, physical reads 0, read-ahead reads 93468, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'Table01Dim'. Scan count 9, logical reads 7688, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'Table02Dim'. Scan count 9, logical reads 7835, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.

 SQL Server Execution Times:
   CPU time = 79406 ms,  elapsed time = 15852 ms.