What Does. Sd Stand for in Data. Table in R

Question

.SD looks useful but I do not really know what I am doing with it. What does it stand for? Why is there a preceding period (full stop). What is happening when I use it?

I read: .SD is a data.table containing the subset of x's data for each group, excluding the group column(s). It can be used when grouping by i, when grouping by by, keyed by, and _ad hoc_ by

Does that mean that the daughter data.tables is held in memory for the next operation?

Accepted Answer

.SD stands for something like "Subset of Data.table". There's no significance to the initial ".", except that it makes it even more unlikely that there will be a clash with a user-defined column name.

If this is your data.table:

DT = data.table(x=rep(c("a","b","c"),each=2), y=c(1,3), v=1:6)
setkey(DT, y)
DT
#    x y v
# 1: a 1 1
# 2: b 1 3
# 3: c 1 5
# 4: a 3 2
# 5: b 3 4
# 6: c 3 6

Doing this may help you see what .SD is:

DT[ , .SD[ , paste(x, v, sep="", collapse="_")], by=y]
#    y       V1
# 1: 1 a1_b3_c5
# 2: 3 a2_b4_c6

Basically, the by=y statement breaks the original data.table into these two sub-data.tables

DT[ , print(.SD), by=y]
# <1st sub-data.table, called '.SD' while it's being operated on>
#    x v
# 1: a 1
# 2: b 3
# 3: c 5
# <2nd sub-data.table, ALSO called '.SD' while it's being operated on>
#    x v
# 1: a 2
# 2: b 4
# 3: c 6
# <final output, since print() doesn't return anything>
# Empty data.table (0 rows) of 1 col: y

and operates on them in turn.

While it is operating on either one, it lets you refer to the current sub-data.table by using the nick-name/handle/symbol .SD. That's very handy, as you can access and operate on the columns just as if you were sitting at the command line working with a single data.table called .SD ... except that here, data.table will carry out those operations on every single sub-data.table defined by combinations of the key, "pasting" them back together and returning the results in a single data.table!

Answer 2

Must Read

Edit:

Given how well-received this answer was, I've converted it into a package vignette now available here

Given how often this comes up, I think this warrants a bit more exposition, beyond the helpful answer given by Josh O'Brien above.

In addition to the Subset of the Data acronym usually cited/created by Josh, I think it's also helpful to consider the "S" to stand for "Selfsame" or "Self-reference" -- .SD is in its most basic guise a reflexive reference to the data.table itself -- as we'll see in examples below, this is particularly helpful for chaining together "queries" (extractions/subsets/etc using [). In particular, this also means that .SD is itself a data.table (with the caveat that it does not allow assignment with :=).

The simpler usage of .SD is for column subsetting (i.e., when .SDcols is specified); I think this version is much more straightforward to understand, so we'll cover that first below. The interpretation of .SD in its second usage, grouping scenarios (i.e., when by = or keyby = is specified), is slightly different, conceptually (though at core it's the same, since, after all, a non-grouped operation is an edge case of grouping with just one group).

Here are some illustrative examples and some other examples of usages that I myself implement often:

Loading Lahman Data

To give this a more real-world feel, rather than making up data, let's load some data sets about baseball from Lahman:

library(data.table) 
library(magrittr) # some piping can be beautiful
library(Lahman)
Teams = as.data.table(Teams)
# *I'm selectively suppressing the printed output of tables here*
Teams
Pitching = as.data.table(Pitching)
# subset for conciseness
Pitching = Pitching[ , .(playerID, yearID, teamID, W, L, G, ERA)]
Pitching

What Does. Sd Stand for in Data. Table in R

3 Answers

Must Read

Edit:

Loading Lahman Data

Sophia Al-Mansoor