Understanding Data Centering in Machine Learning: A Comprehensive Discussion on PCA and Dimensionality Reduction

Sharu introduces herself as the speaker for the session on machine learning techniques.

The session is recorded but not live-streamed for better accessibility.

The session is streamed to enhance accessibility based on a request for better accessibility compared to previous sessions.

There was a discussion on why U transpose V equals the norm of U and V multiplied by cosine Theta, with a request for clarification if needed.

A participant raises a doubt about a practice programming assignment, seeking guidance and sample questions with worked-out solutions for better understanding.

Participants request sample questions with worked-out solutions for numerical problem-solving competence, emphasizing the need for practical examples over theoretical understanding.

Participants express interest in discussing specific questions from graded assignments to enhance problem-solving competence and reference past term materials for guidance.

The speaker mentions a programming assignment that needs to be discussed. They express a need to present their screen for this assignment.

There is an issue with screen sharing as the speaker is unable to share their screen. Others are also facing the same problem, prompting the speaker to suggest that screen sharing might have been disabled.

The assignment being discussed is not graded, and even if the answer is submitted, it only shows the correct answer without any grading.

The speaker expresses a doubt regarding question four of the assignment and seeks clarification. They mention getting an answer of 29 for the question.

The speaker discusses the logic behind the code they wrote for question four. They explain the process of centering the data matrix and the calculation of means along rows and columns.

The speaker requests a review of their code and explains the logic behind it. They mention that the code might be messy due to working on it a year ago.

When dealing with a data matrix, it's crucial to differentiate between arranging data points as columns or rows. If data points are arranged as rows, calculating the mean along the rows is essential to obtain the main data point.

Calculating the mean along a specific axis in a data matrix, denoted as XIs equal to Z, involves averaging the values along that axis. In programming, such as Python, zero-based indexing is commonly used, where mentally associating zero with the axis helps in understanding the mean calculation.

When calculating the mean for features like height and width along XIs equal to one, the result provides an average feature value. For instance, averaging height and width values gives a single column representing the average feature.

The mean calculation for data points involves obtaining the average of data values along a specific axis. If there are multiple data points, the mean result will be a vector or column representing the average values of those data points.

In computing the covariance matrix for a centered data set, it's crucial to subtract the mean data point from the data matrix. This process ensures that the data is centered around the mean data point for accurate covariance calculations.

When calculating the mean along the X-axis, it results in a mean vector, not the mean of the data points. The feature vectors should be chosen carefully to ensure the mean is meaningful, especially when dealing with non-comparable features like height, weight, and categorical variables like color.

Selecting feature vectors where the mean makes sense is crucial. For example, combining height and width to maintain a sensible unit of length ensures the mean of the columns represents an average feature, forming a vector with as many numerical recordings as the number of data points.

To calculate the mean accurately, it's essential to organize the data properly. Taking the column-wise average and subtracting it from the respective features ensures a more meaningful approach than averaging height and weight directly.

Distinguishing between the mean of data points and feature vectors is crucial. The mean data point is the average of rows, while the mean feature vector is the average of feature recordings, highlighting the importance of correctly centering a dataset for accurate analysis.

A student inquired about finding the variance along the first principal component in a graded assignment. The instructor explained that the variance is determined by the dominant eigenvalue in the covariance matrix of the centered dataset. By computing the covariance matrix and identifying the eigenvalues, the variance of each principal component can be obtained. Higher variance along a principal component indicates less collision when projecting data points onto that component, preserving the uniqueness of the original data.

The variance along principal components reflects the informativeness and uniqueness of the data representation. Higher variance implies better fitting of the data points on the component, leading to a more informative and representative representation. It indicates the extent to which the original data's unique characteristics are retained in the reduced dimensional space.

The instructor acknowledged the student's understanding of the variance concept and concluded the discussion on finding variances along principal components. The session transitioned to potential theory presentations or practice questions, inviting further queries from the students. The availability of recordings on YouTube and the course calendar for future reference was also highlighted.

The speaker plans to share the screen and present using slides prepared by Karthik S. They aim to complement the existing material with worked-out examples or similar content to enhance the understanding of the topic.

The speaker mentions that the notes will be shared via a link and asks for confirmation on accessibility.

The concept of representation is explained using the example of condensing hours of footage into a shorter, more concise version while retaining the essence. In statistics, a random sample representing a population is another example of representation, where key statistical aspects are preserved.

In PCA, the goal is to ensure that data points that are far apart in the original 10-dimensional representation remain far apart in the two-dimensional representation, and vice versa for close data points. This spatial interrelation preservation across dimensionality reduction helps create a lower-dimensional representation that retains the essence of the original data.

Representation learning in PCA focuses on creating a lower-dimensional representation that preserves the spatial interrelatedness of data points. It aims to maintain the relationships between points, whether they are far apart or close together, in the reduced dimensionality space.

In machine learning, dimensionality reduction involves converting feature vectors with multiple features into a lower-dimensional space. The process aims to reduce the number of features while retaining essential information for analysis and modeling.

The internal mechanism of dimensionality reduction involves finding the best representation for each data point to minimize the sum of squared errors. This optimization process quantifies the quality of the representation by measuring the Euclidean distance between the original data and its reduced representation.

In the discussion, Mean Square Error (MSE) in Principal Component Analysis (PCA) was highlighted. MSE is calculated as the average of the Euclidean distances between the original data points and their representations. This involves computing the L2 Norm or the Euclidean distance between each original data point Xi and its representation XiR. The goal is to minimize MSE in PCA to ensure that the representations closely match the original data points in a lower-dimensional space.

There was a clarification regarding the difference between variance and Mean Square Error (MSE) in the context of PCA. While variance measures the spread of data points, MSE in PCA focuses on minimizing the error between original data points and their representations. The ideal scenario in PCA is when data points lie in a lower-dimensional plane, leading to representations that closely resemble the original data points.

The discussion compares the compression algorithm PZ with PCA. PZ algorithm compresses data points projected on a 2D plane, resulting in lossless compression if the original data points lie on a 2D structure. In contrast, PCA allows compression for any data set, but it is inherently lossy. The optimization in PCA aims to minimize loss, although some loss is inevitable.

The conversation delves into eigenvalues and data alignment. In the context of a 10-dimensional data set, the largest eigenvalue is used to understand the alignment of the data. By plotting based on this eigenvalue, insights into data alignment are gained, facilitating further analysis.

The dialogue explores the selection of principal components based on eigenvalues. When reducing dimensions to a two-dimensional representation, the principal components corresponding to the largest eigenvalues are retained. This selection process ensures that the most informative components are preserved for effective data representation.

The conversation addresses the quantification of loss in dimensionality reduction. Transitioning from a 3D to a 2D space results in a loss of information due to the reduction in dimensions. The loss is quantified through mathematical expressions, such as the projection of data points onto a subspace and the calculation of distances using norms like the L2 norm.

The discussion highlights the use of standard expressions in dimensionality reduction. Squaring the projection of data points onto a subspace is a common practice to calculate distances, such as the L2 norm. This standard expression is integral to techniques like PCA, which rely on Euclidean distances for analysis and decision-making.

Different choices of Minkowski distance can lead to various dimensionality reduction techniques. The use of L2 distance in this context is highlighted, showcasing how it influences the results obtained.

The discussion delves into the interpretation of squaring in L2 distance calculations. While squaring may not change the positivity of the quantity, it plays a role in ensuring differentiability and convenience in mathematical operations.

The concept of vector addition and norm calculation is explained using a visual analogy of a right angle triangle. The calculation of the L2 norm is demonstrated through vector addition, emphasizing the geometric interpretation of the norm.

A comparison is drawn between the L2 norm and L1 distance, highlighting the difference in calculation methods. L1 distance involves the sum of the absolute components of a vector, while the L2 norm focuses on the square root of the sum of squared components.

The directionality of arrows in the context discussed can be from X2 to X1 or X1 to X2, but the length of the arrow remains the same. The direction may be flipped, but the length stays constant.

The length of the arrow represents a positive quantity. It is emphasized that the length of the arrow is always positive, regardless of its direction.

When discussing the norm, the actual differences between components are squared and added. Taking the norm ensures that the length of the arrow is considered, while just taking the absolute difference may lead to cancellations.

An example with vectors of 3 and 4 is used to illustrate the calculation of the L2 norm. By squaring each component and adding them, the norm of the vectors can be determined, emphasizing the difference between adding vectors directly and adding their norms.

The L2 norm calculation involves squaring each component of a vector and then adding them together. This method ensures that the norm is always positive and provides a clear representation of the vector's magnitude.

The objective is to find representations that are closest to the original data points while being in a lower-dimensional space. The goal is to maintain proximity to the original data while reducing the dimensionality for efficiency.

When representing a data point X on a straight line W, there are three choices: the normal incident on the straight line, dropping a perpendicular from X to W. The normal incident is always the shortest distance from X to W, highlighting the concept of projections. This concept is crucial in understanding the projection of X onto W, as discussed in MLF week three.

In the context of projecting X onto W with a unit vector, X transpose W to W represents the projection of X onto W. The base of the right angle triangle formed is X transpose W to W, with the height being the residual X minus the base. This projection is essential in minimizing the residual between the green and blue orbs, forming the mathematical objective in PCA.

The objective function in PCA aims to minimize the average residual between the green and blue orbs across all data points. This involves finding a set of W vectors that satisfy the minimization criteria. PCA is taught as an iterative process, including the computation of the covariance matrix and sorting it by non-negative eigenvalues.

In PCA, the process of finding principal components is iterative. It starts by determining the first principal component, then constructing residuals from it. The next step involves finding the second principal component by applying the same process to the residuals, not the original vectors. This iterative approach contrasts with a one-shot method, providing a more grounded and insightful understanding of the data.

Vector algebra plays a crucial role in machine learning, particularly in understanding concepts like principal component analysis. While not heavily emphasized in foundational courses, having exposure to vector algebra can enhance competence in machine learning development and communication of ideas. It is recommended to explore vector algebra independently to build a strong foundation for future learning.

The discussion establishes the equivalence between Principal Component Analysis (PCA) and loss minimization. PCA aims to find a low-dimensional substitute (proxy) for original data points that closely represent the data. By minimizing loss, PCA seeks to create proxies that are as close to the original data points as possible, emphasizing the importance of accurate representation in data analysis.

The principal components represent a hyperplane where data points are projected. The goal is to minimize the L2 norm distance between the original data points and their projections on this hyperplane, resulting in the least loss.

Minimizing losses through principal components is mathematically equivalent to maximizing variance. This means choosing principal components that maximize spread and distinctiveness along them, preserving the informativeness and diversity of the original data points.

The concept of minimizing loss in informativeness is equivalent to choosing principal components that maximize variance. This dual perspective highlights the importance of both minimizing distance between original and projected data points and maximizing spread for effective representation.

In MLT, the data points are centered around the origin, unlike in MLF where the data set could be away from the origin. MLF focuses on obtaining lower dimensional proxies, while in MLT, the emphasis is on interrelatedness and representation. Representation involves condensing or filtering a larger dataset to retain essential aspects, such as interrelatedness in PCA.

Although proxies and representations are similar, in MLF, proxies are substitutes for a dataset that may not be centered around the origin, while in MLT, representation focuses on condensing or filtering a larger dataset to retain desired essence, such as interrelatedness in PCA.

In PCA, the essence of interrelatedness between data points is preserved. This means that the spatial relationship of each data point with every other data point is maintained as best as possible, ensuring that the smaller representation retains the essential relationships of the larger dataset.

A water treatment project involving 200 units, each with 50 people, aims to identify areas for improvement and optimization. The project team lacks domain knowledge and is primarily composed of economists seeking to innovate based on insights gathered from key individuals in each unit, rather than surveying all 10,000 individuals directly.

In the discussion, it was mentioned that instead of talking to 50 people in a unit, the approach was to talk to just five or eight people, considered representative enough. This method aimed to minimize the distance between data points, ensuring that the representation closely reflects the original data.

A key distinction highlighted was that in MLF (Machine Learning Framework), the focus was on proxies, where represented data points were shifted to align with the original data cloud. In contrast, in MLT (Machine Learning Theory), the emphasis was solely on representations, disregarding the proximity to the original data points. This shift led to potential loss in interpretability as the represented data points existed in a different frame of reference.

The discussion delved into the importance of maintaining interrelatedness between data points in the representation subspace. It was emphasized that if two pairs of points, A1 A2 and B1 B2, were further apart in the original space, they should also be comparably distant in the representation subspace. This concept aimed to preserve the essence and relationships present in the original data.

In MLT, the focus is on working with centered data points rather than interpreting or making sense of individual data points. The goal is to shift data points to the same space as the original dataset, preserving properties like distance relationships in a lower-dimensional representation.

The Coari Matrix in PCA has an interpretation where loss minimization is equivalent to choosing principal components that maximize the spread of data points. By ensuring that each principal component individually maximizes variance along its axis, the overall spread in the subspace spanned by these components is also maximized.

The PCA algorithm involves mean centering the data, finding the covariance matrix, calculating eigenvalues and eigenvectors, and reverse sorting the eigenvectors based on eigenvalues. The top k eigenvectors are then chosen for a k-dimensional representation of the data.

The discussion delves into the mathematical substantiation behind the PCA algorithm, questioning the origin of steps like taking the covariance matrix and reverse sorting eigenvectors. It emphasizes that these steps are not arbitrary but have a mathematical basis for maximizing variance in the data representation.

The norm of W is set to be equal to one, indicating that it lies around the unit circle for convenience. Principal components are used as spanners to span the axis, not as vectors. Data points can be spread across the plane, and treating the principal component around the unit circle involves projecting onto W.

To span a straight line with a principal component, only a unit vector in the direction of W is needed. No additional components are required beyond a unit vector to span a one-dimensional subspace.

The projection and W are orthogonal, forming a right angle triangle. Visualizing points in 3D projected onto a 2D plane involves finding the point of incidence, which represents the best approximation or residual.

Not taking the norm of W as one does not change the results significantly. The ARG Max remains invariant, even if the norm is not restricted to one. The formula and PC algorithm remain consistent regardless of the unit circle constraint.

On the unit hypersphere, there are infinitely many choices of vectors that maximize a given quantity for a given C. Gradient descent may be used to handle the optimization problem with numerous possible choices of W.

In optimization, the domain is infinite, requiring a set of solutions. Instead of iterative searching, an analytical approach like the quadratic equation formula is used to directly find solutions without manual searching through an infinite set of options.

In optimization, restricting parameters to vectors contained in the unit circle or unit sphere helps in finding maximum candidates efficiently. Without restrictions, the maximum value can increase indefinitely, leading to unbounded problems.

Recorded sessions and YouTube links for the classes will be available on the Google Calendar where the session links were found. The MLT channel also contains resources and recordings for further reference.

The discussion touched upon the importance of gaining competence in vector calculus, particularly in understanding concepts like hyperplanes and hyperspheres in two or more dimensions. Hyper is used as a prefix for general two or more dimensional spaces, with hyperplane being a vector space and hypersphere representing a curved structure. The session emphasized the significance of proficiency in vector calculus for better comprehension of derivations in machine learning, linear algebra, and statistics.

The process of recording and streaming sessions was discussed, highlighting the delay in uploading video recordings due to the time taken by Google to transcode raw files into standard video formats. The speaker mentioned the limitation of Google Drive space for storing heavy MP4 files, leading to potential unavailability of recordings. Alternative methods like sharing streaming links in the discourse forum were suggested for accessibility.

The relevance of vector calculus in the syllabus was debated, with some participants questioning its importance. While acknowledging that it may not be crucial for exams, the speaker emphasized the value of gaining proficiency in vector calculus to enhance understanding of derivations in machine learning. The discussion highlighted the benefits of being comfortable with vector calculus concepts for smoother comprehension of complex mathematical ideas.

Participants discussed accessing Drive links, with some facing difficulties while others were able to access the content. Suggestions were made to try accessing the link from specific accounts and browsers within a designated ecosystem for smoother access. The speaker offered assistance in adding participants to the Drive link to facilitate seamless access to the shared resources.

The discussion delves into the W transpose CW operation, where CW represents a vector to vector mapping. The Matrix C is described as a square matrix, always transforming RN to RN, for example, R3 to R3. The linear transformation represented by Matrix C sends input vector W to an output vector V. The dot product of W transpose V signifies the similarity between the input and output vectors.

The Coari Matrix is highlighted as a measure of similarity between vectors, specifically the dot product between two normalized vectors. It is explained that the dot product serves as a metric for similarity, akin to angular displacement, providing a different perspective compared to spatial distances like L2 or Minkowski distance.

The dot product between two normalized vectors provides a measure of similarity. A dot product of 1 indicates vectors are in the same direction, while 0 means they are perpendicular. This concept is crucial for understanding vector relationships.

By performing the spectral decomposition of matrix C, we obtain W transpose R transpose D R W. This expression plays a significant role in understanding the transformation and basis changes in linear algebra.

Matrix-vector multiplication can be interpreted as applying a linear transformation on the vector, with the matrix representing the transformation. This interpretation is fundamental in understanding the impact of matrix operations on vectors.

Changing the basis grid of vector W to align with the principal components aims to maximize the dot product. This strategy involves selecting the eigenvalues to optimize the dot product, leading to a deeper understanding of vector transformations.

The matrix RW performs a rotation, known as an automorphism, by rotating the basis grid to align with the principal component vectors. This rotation facilitates a clearer representation of vectors in a new basis grid, enhancing the interpretation of linear transformations.

The diagonal matrix in the context of component-wise scaling is discussed. It is explained that the diagonal matrix scales components along principal components in an orthogonal basis grid. This scaling occurs after the components are measured along principal components, resulting in maximum scaling values when the transformation is along diagonal vectors.

The speaker expresses interest in exploring additional aspects related to the diagonal matrix scaling in future sessions. They mention a preference for intuitive storytelling to enhance understanding and visualization. The potential for shedding light on intuition and visualization through the discussion of the igen decomposition and bracketing is highlighted.

The session concludes with plans for the next meeting on Tuesday at 8:35. The speaker expresses gratitude for the discussion and looks forward to future interactions. Access to a drive is requested, and a small doubt is mentioned, showing continued interest and engagement in the topic.